Sep 08 13:16:08.416690 ip-10-0-130-178 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 13:16:08.822536 ip-10-0-130-178 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:16:08.822536 ip-10-0-130-178 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 13:16:08.822536 ip-10-0-130-178 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:16:08.822536 ip-10-0-130-178 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 13:16:08.822536 ip-10-0-130-178 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:16:08.823521 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.823424    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 13:16:08.827106 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827088    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:08.827154 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827107    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:08.827154 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827113    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:08.827154 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827117    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:08.827154 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827121    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:08.827154 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827125    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:08.827154 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827129    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:08.827154 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827132    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:08.827154 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827136    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:08.827154 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827139    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:08.827154 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827143    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:08.827154 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827145    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:08.827154 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827148    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:08.827154 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827151    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:08.827154 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827154    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:08.827154 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827156    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:08.827154 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827159    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:08.827154 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827162    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:08.827588 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827169    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:08.827588 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827172    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:08.827588 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827175    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:08.827588 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827178    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:08.827588 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827180    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:08.827588 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827183    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:08.827588 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827192    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:08.827588 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827194    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:08.827588 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827197    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:08.827588 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827200    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:08.827588 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827202    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:08.827588 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827205    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:08.827588 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827207    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:08.827588 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827210    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:08.827588 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827213    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:08.827588 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827215    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:08.827588 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827218    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:08.827588 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827221    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:08.827588 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827223    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:08.827588 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827226    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:08.828069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827228    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:08.828069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827231    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:08.828069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827233    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:08.828069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827236    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:08.828069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827239    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:08.828069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827242    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:08.828069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827244    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:08.828069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827247    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:08.828069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827249    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:08.828069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827252    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:08.828069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827255    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:08.828069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827257    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:08.828069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827260    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:08.828069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827263    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:08.828069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827266    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:08.828069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827269    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:08.828069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827271    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:08.828069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827274    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:08.828069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827276    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:08.828069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827279    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:08.828575 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827282    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:08.828575 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827294    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:08.828575 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827297    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:08.828575 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827300    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:08.828575 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827303    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:08.828575 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827306    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:08.828575 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827308    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:08.828575 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827311    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:08.828575 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827314    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:08.828575 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827316    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:08.828575 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827319    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:08.828575 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827321    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:08.828575 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827326    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:08.828575 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827329    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:08.828575 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827331    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:08.828575 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827334    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:08.828575 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827337    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:08.828575 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827339    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:08.828575 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827342    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:08.828575 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827344    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:08.829059 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827347    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:08.829059 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827350    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:08.829059 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827353    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:08.829059 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827356    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:08.829059 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827358    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:08.829059 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827362    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:08.829059 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827365    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:08.829059 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827368    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:08.829059 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827370    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:08.829059 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827374    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:08.829059 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827378    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:08.829059 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827381    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:08.829059 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827384    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:08.829059 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827387    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:08.829059 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827390    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:08.829059 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827393    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:08.829059 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827395    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:08.829059 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827398    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:08.829059 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827400    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:08.829059 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827403    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:08.829561 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.827405    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:08.829561 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828190    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:08.829561 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828197    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:08.829561 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828201    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:08.829561 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828204    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:08.829561 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828207    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:08.829561 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828211    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:08.829561 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828214    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:08.829561 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828217    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:08.829561 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828219    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:08.829561 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828223    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:08.829561 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828226    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:08.829561 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828229    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:08.829561 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828232    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:08.829561 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828235    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:08.829561 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828238    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:08.829561 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828241    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:08.829561 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828244    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:08.829561 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828246    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:08.829561 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828249    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:08.830099 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828252    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:08.830099 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828255    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:08.830099 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828257    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:08.830099 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828260    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:08.830099 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828262    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:08.830099 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828265    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:08.830099 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828268    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:08.830099 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828270    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:08.830099 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828273    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:08.830099 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828275    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:08.830099 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828278    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:08.830099 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828280    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:08.830099 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828283    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:08.830099 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828303    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:08.830099 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828306    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:08.830099 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828309    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:08.830099 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828312    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:08.830099 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828315    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:08.830099 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828317    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:08.830586 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828320    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:08.830586 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828323    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:08.830586 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828325    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:08.830586 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828328    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:08.830586 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828331    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:08.830586 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828334    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:08.830586 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828337    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:08.830586 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828340    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:08.830586 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828342    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:08.830586 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828346    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:08.830586 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828350    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:08.830586 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828352    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:08.830586 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828355    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:08.830586 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828358    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:08.830586 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828360    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:08.830586 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828363    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:08.830586 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828366    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:08.830586 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828368    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:08.830586 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828371    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:08.830586 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828374    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:08.831083 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828377    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:08.831083 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828379    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:08.831083 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828382    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:08.831083 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828384    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:08.831083 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828387    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:08.831083 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828390    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:08.831083 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828392    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:08.831083 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828395    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:08.831083 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828397    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:08.831083 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828400    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:08.831083 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828403    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:08.831083 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828409    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:08.831083 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828412    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:08.831083 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828415    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:08.831083 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828418    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:08.831083 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828421    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:08.831083 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828424    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:08.831083 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828427    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:08.831083 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828429    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:08.831083 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828431    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:08.831596 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828434    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:08.831596 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828437    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:08.831596 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828440    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:08.831596 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828443    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:08.831596 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828445    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:08.831596 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828449    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:08.831596 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828452    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:08.831596 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828454    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:08.831596 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828457    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:08.831596 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828459    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:08.831596 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828462    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:08.831596 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828464    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:08.831596 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828467    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:08.831596 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828470    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:08.831596 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828472    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:08.831596 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828474    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:08.831596 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828477    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:08.831596 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828481    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:08.831596 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828484    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:08.831596 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828487    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:08.832077 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828491    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:08.832077 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828879    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:08.832077 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828885    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:08.832077 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828888    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:08.832077 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828891    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:08.832077 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828894    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:08.832077 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828897    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:08.832077 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828900    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:08.832077 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828902    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:08.832077 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828905    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:08.832077 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828908    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:08.832077 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828910    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:08.832077 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828913    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:08.832077 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828916    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:08.832077 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828919    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:08.832077 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828922    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:08.832077 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828925    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:08.832077 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828928    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:08.832077 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828930    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:08.832077 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828933    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:08.832578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828937    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:08.832578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828940    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:08.832578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828943    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:08.832578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828947    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:08.832578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828950    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:08.832578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828953    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:08.832578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828956    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:08.832578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828959    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:08.832578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828961    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:08.832578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828964    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:08.832578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828966    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:08.832578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828968    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:08.832578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828971    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:08.832578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828974    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:08.832578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828976    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:08.832578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828979    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:08.832578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828982    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:08.832578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828984    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:08.832578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828987    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:08.833061 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828990    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:08.833061 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828993    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:08.833061 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828995    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:08.833061 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.828998    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:08.833061 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829001    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:08.833061 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829003    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:08.833061 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829006    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:08.833061 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829008    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:08.833061 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829012    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:08.833061 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829015    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:08.833061 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829017    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:08.833061 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829020    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:08.833061 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829022    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:08.833061 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829025    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:08.833061 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829027    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:08.833061 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829030    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:08.833061 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829032    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:08.833061 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829035    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:08.833061 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829037    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:08.833061 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829040    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:08.833578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829042    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:08.833578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829045    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:08.833578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829047    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:08.833578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829050    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:08.833578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829053    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:08.833578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829056    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:08.833578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829059    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:08.833578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829062    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:08.833578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829065    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:08.833578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829068    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:08.833578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829071    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:08.833578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829073    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:08.833578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829076    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:08.833578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829078    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:08.833578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829081    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:08.833578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829084    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:08.833578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829087    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:08.833578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829089    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:08.833578 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829092    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:08.834051 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829094    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:08.834051 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829097    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:08.834051 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829100    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:08.834051 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829103    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:08.834051 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829105    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:08.834051 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829108    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:08.834051 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829110    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:08.834051 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829113    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:08.834051 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829115    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:08.834051 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829118    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:08.834051 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829121    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:08.834051 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829123    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:08.834051 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829126    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:08.834051 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829128    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:08.834051 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829130    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:08.834051 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829133    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:08.834051 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829135    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:08.834051 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829138    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:08.834051 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829140    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:08.834051 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829143    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829146    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829149    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829226    2561 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829270    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829278    2561 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829282    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829299    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829303    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829312    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829317    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829320    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829323    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829327    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829330    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829334    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829338    2561 flags.go:64] FLAG: --cgroup-root=""
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829341    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829344    2561 flags.go:64] FLAG: --client-ca-file=""
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829347    2561 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829350    2561 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829355    2561 flags.go:64] FLAG: --cluster-domain=""
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829358    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 13:16:08.834695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829361    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829364    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829367    2561 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829371    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829374    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829377    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829381    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829384    2561 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829388    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829391    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829395    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829398    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829402    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829405    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829408    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829411    2561 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829414    2561 flags.go:64] FLAG: --enable-server="true"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829418    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829422    2561 flags.go:64] FLAG: --event-burst="100"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829425    2561 flags.go:64] FLAG: --event-qps="50"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829428    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829431    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829435    2561 flags.go:64] FLAG: --eviction-hard=""
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829438    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 13:16:08.835262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829441    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829444    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829448    2561 flags.go:64] FLAG: --eviction-soft=""
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829451    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829454    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829457    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829460    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829463    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829468    2561 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829471    2561 flags.go:64] FLAG: --feature-gates=""
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829478    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829482    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829486    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829489    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829492    2561 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829495    2561 flags.go:64] FLAG: --help="false"
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829498    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-130-178.ec2.internal"
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829501    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829504    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829508    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829511    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829515    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829518    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 13:16:08.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829520    2561 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829524    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829526    2561 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829530    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829533    2561 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829536    2561 flags.go:64] FLAG: --kube-reserved=""
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829539    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829542    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829545    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829547    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829550    2561 flags.go:64] FLAG: --lock-file=""
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829553    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829556    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829559    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829565    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829568    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829571    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829575    2561 flags.go:64] FLAG: --logging-format="text"
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829578    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829583    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829586    2561 flags.go:64] FLAG: --manifest-url=""
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829589    2561 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829593    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829596    2561 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829601    2561 flags.go:64] FLAG: --max-pods="110"
Sep 08 13:16:08.836505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829604    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829607    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829610    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829613    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829616    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829619    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829622    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829630    2561 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829633    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829636    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829640    2561 flags.go:64] FLAG: --pod-cidr=""
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829643    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829649    2561 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829652    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829655    2561 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829658    2561 flags.go:64] FLAG: --port="10250"
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829661    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829664    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-087a8fddb1e394aaa"
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829667    2561 flags.go:64] FLAG: --qos-reserved=""
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829670    2561 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829678    2561 flags.go:64] FLAG: --register-node="true"
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829681    2561 flags.go:64] FLAG: --register-with-taints=""
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829703    2561 flags.go:64] FLAG: --registry-burst="10"
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829708    2561 flags.go:64] FLAG: --registry-qps="5"
Sep 08 13:16:08.837169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829713    2561 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829716    2561 flags.go:64] FLAG: --reserved-memory=""
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829720    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829725    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829728    2561 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829731    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829734    2561 flags.go:64] FLAG: --runonce="false"
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829737    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829740    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829743    2561 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829746    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829749    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829752    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829755    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829759    2561 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829762    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829765    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829768    2561 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829770    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829773    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829776    2561 flags.go:64] FLAG: --system-cgroups=""
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829779    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829785    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829788    2561 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829791    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 13:16:08.837968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829795    2561 flags.go:64] FLAG: --tls-min-version=""
Sep 08 13:16:08.838987 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829798    2561 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 13:16:08.838987 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829801    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 13:16:08.838987 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829803    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 13:16:08.838987 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829807    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 13:16:08.838987 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829809    2561 flags.go:64] FLAG: --v="2"
Sep 08 13:16:08.838987 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829813    2561 flags.go:64] FLAG: --version="false"
Sep 08 13:16:08.838987 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829819    2561 flags.go:64] FLAG: --vmodule=""
Sep 08 13:16:08.838987 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829822    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 13:16:08.838987 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.829826    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 13:16:08.838987 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829946    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:08.838987 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829950    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:08.838987 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829953    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:08.838987 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829956    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:08.838987 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829961    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:08.838987 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829964    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:08.838987 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829967    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:08.838987 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829970    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:08.838987 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829973    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:08.838987 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829976    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:08.838987 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829979    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:08.838987 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829982    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:08.839591 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829985    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:08.839591 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829988    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:08.839591 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829991    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:08.839591 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829994    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:08.839591 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.829997    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:08.839591 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830000    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:08.839591 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830002    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:08.839591 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830005    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:08.839591 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830007    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:08.839591 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830010    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:08.839591 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830013    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:08.839591 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830015    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:08.839591 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830018    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:08.839591 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830020    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:08.839591 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830023    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:08.839591 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830026    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:08.839591 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830029    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:08.839591 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830033    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:08.839591 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830036    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:08.840057 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830040    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:08.840057 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830044    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:08.840057 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830047    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:08.840057 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830049    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:08.840057 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830052    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:08.840057 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830054    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:08.840057 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830057    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:08.840057 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830060    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:08.840057 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830062    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:08.840057 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830065    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:08.840057 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830067    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:08.840057 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830070    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:08.840057 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830072    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:08.840057 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830075    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:08.840057 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830078    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:08.840057 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830081    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:08.840057 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830083    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:08.840057 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830086    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:08.840057 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830089    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:08.840057 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830091    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:08.840571 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830093    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:08.840571 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830096    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:08.840571 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830099    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:08.840571 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830101    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:08.840571 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830104    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:08.840571 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830107    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:08.840571 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830109    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:08.840571 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830112    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:08.840571 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830114    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:08.840571 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830117    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:08.840571 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830121    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:08.840571 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830125    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:08.840571 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830127    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:08.840571 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830132    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:08.840571 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830135    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:08.840571 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830138    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:08.840571 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830141    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:08.840571 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830144    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:08.840571 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830147    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:08.841042 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830149    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:08.841042 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830152    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:08.841042 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830154    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:08.841042 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830157    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:08.841042 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830159    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:08.841042 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830162    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:08.841042 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830165    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:08.841042 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830167    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:08.841042 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830170    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:08.841042 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830173    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:08.841042 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830175    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:08.841042 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830177    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:08.841042 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830180    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:08.841042 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830182    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:08.841042 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830185    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:08.841042 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830188    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:08.841042 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830190    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:08.841042 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830193    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:08.841042 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830195    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:08.841042 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830198    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:08.841558 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830200    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:08.841558 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830203    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:08.841558 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830206    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:08.841558 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830210    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:08.841558 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830213    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:08.841558 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830216    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:08.841558 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830219    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:08.841558 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830221    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:08.841558 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.830224    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:08.841558 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.831132    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:16:08.841558 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.839618    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 13:16:08.841558 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.839636    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 13:16:08.841558 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.840434    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 13:16:08.841558 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.840446    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 13:16:08.841558 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840507    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:08.841934 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840513    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:08.841934 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840518    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:08.841934 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840522    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:08.841934 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840526    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:08.841934 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840529    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:08.841934 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840532    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:08.841934 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840535    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:08.841934 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840538    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:08.841934 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840541    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:08.841934 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840543    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:08.841934 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840547    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:08.841934 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840549    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:08.841934 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840552    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:08.841934 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840555    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:08.841934 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840557    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:08.841934 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840560    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:08.841934 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840563    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:08.841934 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840566    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:08.841934 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840569    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:08.841934 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840571    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:08.842585 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840574    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:08.842585 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840577    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:08.842585 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840580    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:08.842585 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840583    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:08.842585 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840586    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:08.842585 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840588    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:08.842585 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840591    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:08.842585 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840594    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:08.842585 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840598    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:08.842585 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840601    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:08.842585 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840604    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:08.842585 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840606    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:08.842585 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840609    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:08.842585 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840612    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:08.842585 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840615    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:08.842585 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840617    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:08.842585 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840620    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:08.842585 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840623    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:08.842585 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840625    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:08.842585 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840628    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:08.843074 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840631    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:08.843074 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840633    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:08.843074 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840636    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:08.843074 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840639    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:08.843074 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840641    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:08.843074 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840644    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:08.843074 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840647    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:08.843074 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840649    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:08.843074 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840652    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:08.843074 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840655    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:08.843074 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840658    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:08.843074 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840660    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:08.843074 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840663    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:08.843074 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840665    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:08.843074 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840668    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:08.843074 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840671    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:08.843074 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840673    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:08.843074 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840676    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:08.843074 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840678    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:08.843555 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840683    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:08.843555 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840687    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:08.843555 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840690    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:08.843555 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840693    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:08.843555 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840696    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:08.843555 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840699    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:08.843555 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840702    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:08.843555 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840706    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:08.843555 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840709    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:08.843555 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840712    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:08.843555 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840715    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:08.843555 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840718    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:08.843555 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840720    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:08.843555 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840723    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:08.843555 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840727    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:08.843555 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840730    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:08.843555 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840733    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:08.843555 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840736    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:08.843555 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840738    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:08.844069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840742    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:08.844069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840745    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:08.844069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840748    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:08.844069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840751    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:08.844069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840754    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:08.844069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840757    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:08.844069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840760    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:08.844069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840762    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:08.844069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840765    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:08.844069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840768    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:08.844069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840771    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:08.844069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840774    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:08.844069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840776    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:08.844069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840780    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:08.844069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840783    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:08.844069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840786    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:08.844069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840788    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:08.844069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840791    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:08.844069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840794    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:08.844069 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840796    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:08.844651 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.840801    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:16:08.844651 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840919    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:08.844651 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840925    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:08.844651 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840928    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:08.844651 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840931    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:08.844651 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840934    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:08.844651 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840937    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:08.844651 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840940    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:08.844651 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840943    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:08.844651 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840946    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:08.844651 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840949    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:08.844651 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840952    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:08.844651 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840954    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:08.844651 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840958    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:08.844651 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840961    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:08.845021 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840964    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:08.845021 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840966    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:08.845021 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840969    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:08.845021 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840973    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:08.845021 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840976    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:08.845021 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840978    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:08.845021 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840981    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:08.845021 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840984    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:08.845021 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840986    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:08.845021 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840990    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:08.845021 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840993    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:08.845021 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840995    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:08.845021 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.840998    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:08.845021 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841001    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:08.845021 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841004    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:08.845021 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841006    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:08.845021 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841009    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:08.845021 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841011    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:08.845021 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841014    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:08.845021 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841016    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:08.845640 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841019    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:08.845640 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841021    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:08.845640 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841024    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:08.845640 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841027    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:08.845640 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841029    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:08.845640 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841032    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:08.845640 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841035    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:08.845640 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841037    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:08.845640 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841040    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:08.845640 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841043    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:08.845640 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841046    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:08.845640 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841050    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:08.845640 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841053    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:08.845640 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841055    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:08.845640 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841058    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:08.845640 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841061    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:08.845640 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841064    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:08.845640 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841066    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:08.845640 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841069    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:08.846102 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841071    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:08.846102 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841074    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:08.846102 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841076    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:08.846102 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841079    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:08.846102 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841081    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:08.846102 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841084    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:08.846102 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841086    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:08.846102 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841089    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:08.846102 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841092    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:08.846102 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841094    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:08.846102 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841097    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:08.846102 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841099    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:08.846102 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841102    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:08.846102 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841104    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:08.846102 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841107    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:08.846102 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841109    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:08.846102 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841112    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:08.846102 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841115    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:08.846102 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841117    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:08.846102 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841120    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:08.846630 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841123    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:08.846630 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841125    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:08.846630 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841128    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:08.846630 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841130    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:08.846630 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841133    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:08.846630 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841136    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:08.846630 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841139    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:08.846630 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841141    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:08.846630 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841144    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:08.846630 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841148    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:08.846630 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841152    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:08.846630 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841154    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:08.846630 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841157    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:08.846630 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841160    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:08.846630 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841162    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:08.846630 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841165    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:08.846630 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841167    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:08.846630 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841170    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:08.846630 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841172    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:08.847088 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841176    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:08.847088 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841180    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:08.847088 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841184    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:08.847088 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841187    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:08.847088 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841190    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:08.847088 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841192    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:08.847088 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:08.841195    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:08.847088 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.841200    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:16:08.847088 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.841357    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 13:16:08.847088 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.843790    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 13:16:08.847088 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.845190    2561 server.go:1019] "Starting client certificate rotation"
Sep 08 13:16:08.847088 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.845321    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 13:16:08.847088 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.845376    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 13:16:08.847814 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.847798    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 13:16:08.851908 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.851884    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 13:16:08.861647 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.861623    2561 log.go:25] "Validated CRI v1 runtime API"
Sep 08 13:16:08.864638 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.864622    2561 log.go:25] "Validated CRI v1 image API"
Sep 08 13:16:08.865465 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.865451    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 13:16:08.867626 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.867606    2561 fs.go:135] Filesystem UUIDs: map[0bc3c49f-39b4-4677-9ef3-cf59621a79d0:/dev/nvme0n1p4 5ba74e81-88a1-4fdd-b07a-6417b3622572:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 08 13:16:08.867693 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.867625    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 13:16:08.872313 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.872282    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 13:16:08.873951 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.873816    2561 manager.go:216] Machine: {Timestamp:2026-09-08 13:16:08.871774211 +0000 UTC m=+0.332419012 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099654 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26d92e27a109b813881769ad09a388 SystemUUID:ec26d92e-27a1-09b8-1388-1769ad09a388 BootID:022be4a6-1d2f-49b0-a19a-b1a3c1a70a3f Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:54:a6:69:2c:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:54:a6:69:2c:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:b8:6e:b5:43:95 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 13:16:08.873951 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.873950    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 13:16:08.874072 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.874045    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 13:16:08.878390 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.878364    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 13:16:08.878576 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.878401    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-178.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 13:16:08.879185 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.879175    2561 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 13:16:08.879222 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.879187    2561 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 13:16:08.879222 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.879196    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 13:16:08.880077 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.880065    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 13:16:08.880865 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.880854    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 13:16:08.883720 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.883707    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 13:16:08.883853 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.883843    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 13:16:08.883893 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.883880    2561 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 13:16:08.884203 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.884190    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 13:16:08.884271 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.884216    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 13:16:08.884271 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.884228    2561 kubelet.go:398] "Adding apiserver pod source"
Sep 08 13:16:08.885118 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.885095    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 13:16:08.885814 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.885751    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 13:16:08.885814 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.885782    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 13:16:08.888082 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.888061    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 13:16:08.890329 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.890282    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 13:16:08.890388 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.890355    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 13:16:08.891498 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.891487    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 13:16:08.891536 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.891504    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 13:16:08.891536 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.891510    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 13:16:08.891536 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.891516    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 13:16:08.891536 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.891524    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 13:16:08.891536 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.891530    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 13:16:08.891666 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.891543    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 13:16:08.891666 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.891548    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 13:16:08.891666 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.891556    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 13:16:08.891666 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.891561    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 13:16:08.891666 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.891570    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 13:16:08.891666 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.891575    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 13:16:08.892378 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.892369    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 13:16:08.892378 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.892378    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 13:16:08.896165 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.896151    2561 server.go:1268] "Started kubelet"
Sep 08 13:16:08.896254 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.896223    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 13:16:08.896303 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.896236    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 13:16:08.896352 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.896339    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 13:16:08.897305 ip-10-0-130-178 systemd[1]: Started Kubernetes Kubelet.
Sep 08 13:16:08.897616 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.897312    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 13:16:08.897714 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.897698    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 13:16:08.897762 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.897734    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9clh9"
Sep 08 13:16:08.898039 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:08.898017    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 13:16:08.898112 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:08.898020    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 13:16:08.901231 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.901215    2561 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 13:16:08.901397 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:08.901375    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 13:16:08.901491 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:08.900416    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-178.ec2.internal.18d35ad9e535b64b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-178.ec2.internal,UID:ip-10-0-130-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-178.ec2.internal,},FirstTimestamp:2026-09-08 13:16:08.896124491 +0000 UTC m=+0.356769292,LastTimestamp:2026-09-08 13:16:08.896124491 +0000 UTC m=+0.356769292,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-178.ec2.internal,}"
Sep 08 13:16:08.903340 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.903321    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 13:16:08.903633 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.903608    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 13:16:08.903814 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:08.903797    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-178.ec2.internal\" not found"
Sep 08 13:16:08.904329 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.904311    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 13:16:08.904425 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.904337    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 13:16:08.904425 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.904349    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 13:16:08.904525 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.904456    2561 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 13:16:08.904525 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.904465    2561 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 13:16:08.904753 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.904734    2561 factory.go:55] Registering systemd factory
Sep 08 13:16:08.904753 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.904754    2561 factory.go:223] Registration of the systemd container factory successfully
Sep 08 13:16:08.905003 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.904986    2561 factory.go:153] Registering CRI-O factory
Sep 08 13:16:08.905051 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.905007    2561 factory.go:223] Registration of the crio container factory successfully
Sep 08 13:16:08.905081 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.905059    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 13:16:08.905113 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.905080    2561 factory.go:103] Registering Raw factory
Sep 08 13:16:08.905113 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.905098    2561 manager.go:1196] Started watching for new ooms in manager
Sep 08 13:16:08.905505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.905488    2561 manager.go:349] Starting recovery of all containers
Sep 08 13:16:08.907010 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.906986    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9clh9"
Sep 08 13:16:08.907211 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:08.907189    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 13:16:08.907344 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:08.907322    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 13:16:08.915128 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.915102    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 13:16:08.915983 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.915963    2561 manager.go:354] Recovery completed
Sep 08 13:16:08.920968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.920953    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:08.923061 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.923040    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:08.923147 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.923078    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:08.923147 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.923092    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:08.923826 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.923680    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:08.923826 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.923680    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 13:16:08.923826 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.923714    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 13:16:08.923826 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.923732    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 13:16:08.925025 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.925011    2561 policy_none.go:49] "None policy: Start"
Sep 08 13:16:08.925101 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.925034    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 13:16:08.925101 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.925047    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 13:16:08.925615 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:08.925524    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-178.ec2.internal.18d35ad9e6d0bb32  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-178.ec2.internal,UID:ip-10-0-130-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-178.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-178.ec2.internal,},FirstTimestamp:2026-09-08 13:16:08.923061042 +0000 UTC m=+0.383705846,LastTimestamp:2026-09-08 13:16:08.923061042 +0000 UTC m=+0.383705846,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-178.ec2.internal,}"
Sep 08 13:16:08.925773 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.925753    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:08.925847 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.925791    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:08.925847 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.925808    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:08.926516 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.926386    2561 policy_none.go:47] "Start"
Sep 08 13:16:08.951100 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.951082    2561 manager.go:337] "Starting Device Plugin manager"
Sep 08 13:16:08.951212 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:08.951128    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 13:16:08.951212 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.951141    2561 server.go:87] "Starting device plugin registration server"
Sep 08 13:16:08.951443 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.951429    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 13:16:08.951535 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.951447    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 13:16:08.951665 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.951646    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 13:16:08.951763 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.951732    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 13:16:08.951763 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.951741    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 13:16:08.953427 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:08.952229    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 13:16:08.953427 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:08.952311    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-178.ec2.internal\" not found"
Sep 08 13:16:08.999841 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.999809    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 13:16:09.000005 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.999872    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 13:16:09.000005 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:08.999907    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 13:16:09.000102 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:09.000019    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 13:16:09.002091 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.002071    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:09.052365 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.052328    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:09.053496 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.053476    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:09.053628 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.053511    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:09.053628 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.053526    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:09.053628 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.053558    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-178.ec2.internal"
Sep 08 13:16:09.061666 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.061644    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-178.ec2.internal"
Sep 08 13:16:09.061767 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:09.061669    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-178.ec2.internal\": node \"ip-10-0-130-178.ec2.internal\" not found"
Sep 08 13:16:09.077590 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:09.077488    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-178.ec2.internal\" not found"
Sep 08 13:16:09.100622 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.100576    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-178.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-178.ec2.internal"]
Sep 08 13:16:09.100761 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.100708    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:09.101794 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.101774    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:09.101898 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.101812    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:09.101898 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.101828    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:09.103824 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.103810    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:09.103882 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.103865    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-178.ec2.internal"
Sep 08 13:16:09.103930 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.103901    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:09.104648 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.104628    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:09.104753 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.104664    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:09.104753 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.104679    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:09.104753 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.104628    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:09.104753 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.104725    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:09.104753 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.104739    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:09.105955 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.105937    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-178.ec2.internal"
Sep 08 13:16:09.105955 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.105943    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b0b12b2fbd9460cffd368b1bb1ef73f5-config\") pod \"kube-apiserver-proxy-ip-10-0-130-178.ec2.internal\" (UID: \"b0b12b2fbd9460cffd368b1bb1ef73f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-178.ec2.internal"
Sep 08 13:16:09.106173 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.105969    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:09.106173 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.106024    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dd5a51648b67abc20f8ee816314baeef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-178.ec2.internal\" (UID: \"dd5a51648b67abc20f8ee816314baeef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-178.ec2.internal"
Sep 08 13:16:09.106173 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.106085    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd5a51648b67abc20f8ee816314baeef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-178.ec2.internal\" (UID: \"dd5a51648b67abc20f8ee816314baeef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-178.ec2.internal"
Sep 08 13:16:09.106683 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.106667    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:09.106804 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.106695    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:09.106804 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.106707    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:09.135141 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:09.135108    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-178.ec2.internal\" not found" node="ip-10-0-130-178.ec2.internal"
Sep 08 13:16:09.139646 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:09.139627    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-178.ec2.internal\" not found" node="ip-10-0-130-178.ec2.internal"
Sep 08 13:16:09.178441 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:09.178408    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-178.ec2.internal\" not found"
Sep 08 13:16:09.206614 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.206584    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dd5a51648b67abc20f8ee816314baeef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-178.ec2.internal\" (UID: \"dd5a51648b67abc20f8ee816314baeef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-178.ec2.internal"
Sep 08 13:16:09.206734 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.206617    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd5a51648b67abc20f8ee816314baeef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-178.ec2.internal\" (UID: \"dd5a51648b67abc20f8ee816314baeef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-178.ec2.internal"
Sep 08 13:16:09.206734 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.206637    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b0b12b2fbd9460cffd368b1bb1ef73f5-config\") pod \"kube-apiserver-proxy-ip-10-0-130-178.ec2.internal\" (UID: \"b0b12b2fbd9460cffd368b1bb1ef73f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-178.ec2.internal"
Sep 08 13:16:09.206734 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.206669    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd5a51648b67abc20f8ee816314baeef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-178.ec2.internal\" (UID: \"dd5a51648b67abc20f8ee816314baeef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-178.ec2.internal"
Sep 08 13:16:09.206734 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.206681    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dd5a51648b67abc20f8ee816314baeef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-178.ec2.internal\" (UID: \"dd5a51648b67abc20f8ee816314baeef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-178.ec2.internal"
Sep 08 13:16:09.206734 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.206704    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b0b12b2fbd9460cffd368b1bb1ef73f5-config\") pod \"kube-apiserver-proxy-ip-10-0-130-178.ec2.internal\" (UID: \"b0b12b2fbd9460cffd368b1bb1ef73f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-178.ec2.internal"
Sep 08 13:16:09.278790 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:09.278746    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-178.ec2.internal\" not found"
Sep 08 13:16:09.379446 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:09.379364    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-178.ec2.internal\" not found"
Sep 08 13:16:09.437008 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.436969    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-178.ec2.internal"
Sep 08 13:16:09.442977 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.442795    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-178.ec2.internal"
Sep 08 13:16:09.479766 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:09.479733    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-178.ec2.internal\" not found"
Sep 08 13:16:09.585173 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:09.585115    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-178.ec2.internal\" not found"
Sep 08 13:16:09.689917 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:09.689816    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-178.ec2.internal\" not found"
Sep 08 13:16:09.790272 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:09.790227    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-178.ec2.internal\" not found"
Sep 08 13:16:09.846875 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.846836    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 13:16:09.847467 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.847041    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 13:16:09.890684 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:09.890633    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-178.ec2.internal\" not found"
Sep 08 13:16:09.912100 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.912050    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 13:11:09 +0000 UTC" deadline="2028-04-10 06:52:59.725942878 +0000 UTC"
Sep 08 13:16:09.912100 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:09.912091    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13913h36m49.813855526s"
Sep 08 13:16:09.993085 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:09.992994    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-178.ec2.internal\" not found"
Sep 08 13:16:10.070188 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.070154    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:10.104778 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.104748    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-178.ec2.internal"
Sep 08 13:16:10.127860 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.127836    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 13:16:10.129922 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.129901    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-178.ec2.internal"
Sep 08 13:16:10.141500 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.141479    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 13:16:10.200125 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.200096    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:10.245861 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.245784    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:10.361433 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:10.361395    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5a51648b67abc20f8ee816314baeef.slice/crio-9533427aa91a2307db0e6df3f4318a0a673be61a86f9b6da2a4eba6b785c481e WatchSource:0}: Error finding container 9533427aa91a2307db0e6df3f4318a0a673be61a86f9b6da2a4eba6b785c481e: Status 404 returned error can't find the container with id 9533427aa91a2307db0e6df3f4318a0a673be61a86f9b6da2a4eba6b785c481e
Sep 08 13:16:10.364887 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.364869    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:16:10.580345 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:10.580311    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b12b2fbd9460cffd368b1bb1ef73f5.slice/crio-85df0a41731d9cc8cf463a5f41e52ca929806cbc0da5de49b0b899e8933b1ad6 WatchSource:0}: Error finding container 85df0a41731d9cc8cf463a5f41e52ca929806cbc0da5de49b0b899e8933b1ad6: Status 404 returned error can't find the container with id 85df0a41731d9cc8cf463a5f41e52ca929806cbc0da5de49b0b899e8933b1ad6
Sep 08 13:16:10.668378 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.668340    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:10.888806 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.888711    2561 apiserver.go:52] "Watching apiserver"
Sep 08 13:16:10.895297 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.895270    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 13:16:10.896224 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.896197    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-t8k5r","openshift-multus/multus-additional-cni-plugins-6f6nt","openshift-network-diagnostics/network-check-target-t27j2","openshift-network-operator/iptables-alerter-tjwr7","openshift-ovn-kubernetes/ovnkube-node-mvnln","kube-system/global-pull-secret-syncer-ktjbq","kube-system/konnectivity-agent-ql8vc","kube-system/kube-apiserver-proxy-ip-10-0-130-178.ec2.internal","openshift-cluster-node-tuning-operator/tuned-v25cf","openshift-image-registry/node-ca-f4gjd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-178.ec2.internal","openshift-multus/multus-n4r6n","openshift-multus/network-metrics-daemon-gsgmd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"]
Sep 08 13:16:10.897929 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.897908    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ql8vc"
Sep 08 13:16:10.899340 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.899316    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:10.899448 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:10.899384    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gsgmd" podUID="246d684e-5ff4-49bb-b39c-660e46385842"
Sep 08 13:16:10.900415 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.900389    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:10.900529 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:10.900485    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t27j2" podUID="8562e28b-ea18-46bb-98e1-5ea840db0ef5"
Sep 08 13:16:10.900643 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.900625    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wqcfl\""
Sep 08 13:16:10.900823 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.900800    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 13:16:10.900919 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.900902    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 13:16:10.901651 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.901627    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tjwr7"
Sep 08 13:16:10.903166 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.903144    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:10.903739 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.903721    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 13:16:10.904055 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.904039    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 13:16:10.904128 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.904089    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zf4jc\""
Sep 08 13:16:10.904191 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.904126    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 13:16:10.904191 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.904138    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:16:10.904596 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.904577    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:10.904688 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:10.904652    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktjbq" podUID="1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e"
Sep 08 13:16:10.905767 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.905744    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 13:16:10.905767 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.905754    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 13:16:10.906091 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.906026    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 13:16:10.906176 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.906124    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 13:16:10.906176 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.906156    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 13:16:10.906291 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.906230    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:10.906291 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.906250    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wgq8q\""
Sep 08 13:16:10.906291 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.906259    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 13:16:10.907682 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.907667    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t8k5r"
Sep 08 13:16:10.909011 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.908994    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:16:10.909404 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.909375    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5fg24\""
Sep 08 13:16:10.909517 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.909492    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 13:16:10.909896 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.909812    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:10.912443 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.912107    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f4gjd"
Sep 08 13:16:10.914471 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.914454    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:10.915038 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.915006    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 13:16:10.915340 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.915257    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 13:16:10.915340 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.915325    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 13:16:10.915520 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.915383    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 13:16:10.915520 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.915455    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 13:16:10.915520 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.915502    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 13:16:10.915671 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.915553    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 13:16:10.915671 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.915601    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 13:16:10.915943 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.915812    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rlhvg\""
Sep 08 13:16:10.915943 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.915824    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 13:16:10.915943 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.915888    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9db9d\""
Sep 08 13:16:10.915943 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.915923    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 13:16:10.916192 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.915993    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:10.916192 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.916083    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pbcsr\""
Sep 08 13:16:10.917087 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917068    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-host-slash\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:10.917166 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917091    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-host-run-netns\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:10.917166 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917108    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-env-overrides\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:10.917166 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917124    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-ovnkube-script-lib\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:10.917166 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917144    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:10.917166 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917162    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq7mj\" (UniqueName: \"kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj\") pod \"network-check-target-t27j2\" (UID: \"8562e28b-ea18-46bb-98e1-5ea840db0ef5\") " pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:10.917375 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917183    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/987c2846-77e0-499e-9765-0aeb9f290255-iptables-alerter-script\") pod \"iptables-alerter-tjwr7\" (UID: \"987c2846-77e0-499e-9765-0aeb9f290255\") " pod="openshift-network-operator/iptables-alerter-tjwr7"
Sep 08 13:16:10.917375 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917216    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/987c2846-77e0-499e-9765-0aeb9f290255-host-slash\") pod \"iptables-alerter-tjwr7\" (UID: \"987c2846-77e0-499e-9765-0aeb9f290255\") " pod="openshift-network-operator/iptables-alerter-tjwr7"
Sep 08 13:16:10.917375 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917236    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-run-openvswitch\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:10.917375 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917253    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdh9v\" (UniqueName: \"kubernetes.io/projected/246d684e-5ff4-49bb-b39c-660e46385842-kube-api-access-rdh9v\") pod \"network-metrics-daemon-gsgmd\" (UID: \"246d684e-5ff4-49bb-b39c-660e46385842\") " pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:10.917375 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917281    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-host-kubelet\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:10.917375 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917303    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-etc-openvswitch\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:10.917375 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917325    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-node-log\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:10.917375 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917349    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-host-run-ovn-kubernetes\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:10.917663 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917386    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-host-cni-netd\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:10.917663 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917403    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-ovn-node-metrics-cert\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:10.917663 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917418    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-kubelet-config\") pod \"global-pull-secret-syncer-ktjbq\" (UID: \"1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e\") " pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:10.917663 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917434    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-var-lib-openvswitch\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:10.917663 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917459    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-log-socket\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:10.917663 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917484    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-dbus\") pod \"global-pull-secret-syncer-ktjbq\" (UID: \"1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e\") " pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:10.917663 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917513    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/53ffac54-50e1-4c1e-97b1-b18f1164f56d-agent-certs\") pod \"konnectivity-agent-ql8vc\" (UID: \"53ffac54-50e1-4c1e-97b1-b18f1164f56d\") " pod="kube-system/konnectivity-agent-ql8vc"
Sep 08 13:16:10.917663 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917543    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/53ffac54-50e1-4c1e-97b1-b18f1164f56d-konnectivity-ca\") pod \"konnectivity-agent-ql8vc\" (UID: \"53ffac54-50e1-4c1e-97b1-b18f1164f56d\") " pod="kube-system/konnectivity-agent-ql8vc"
Sep 08 13:16:10.917663 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917568    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs\") pod \"network-metrics-daemon-gsgmd\" (UID: \"246d684e-5ff4-49bb-b39c-660e46385842\") " pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:10.917663 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917583    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-run-systemd\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:10.917663 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917596    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-host-cni-bin\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:10.917663 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917612    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-ovnkube-config\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:10.918088 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917665    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-systemd-units\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:10.918088 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917722    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-run-ovn\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:10.918088 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917748    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9k8r\" (UniqueName: \"kubernetes.io/projected/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-kube-api-access-s9k8r\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:10.918088 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret\") pod \"global-pull-secret-syncer-ktjbq\" (UID: \"1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e\") " pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:10.918088 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.917817    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9z9f\" (UniqueName: \"kubernetes.io/projected/987c2846-77e0-499e-9765-0aeb9f290255-kube-api-access-f9z9f\") pod \"iptables-alerter-tjwr7\" (UID: \"987c2846-77e0-499e-9765-0aeb9f290255\") " pod="openshift-network-operator/iptables-alerter-tjwr7"
Sep 08 13:16:10.920243 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.920226    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 13:16:10.920552 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.920533    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 13:16:10.920659 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.920632    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 13:16:10.920768 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.920658    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h2ngk\""
Sep 08 13:16:10.921221 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.921206    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 13:16:10.922109 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.922053    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g99sc\""
Sep 08 13:16:10.926857 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.926835    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 13:16:10.948001 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.947972    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6525f"
Sep 08 13:16:10.959949 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:10.959929    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6525f"
Sep 08 13:16:11.005189 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.005159    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 13:16:11.005381 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.005270    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-178.ec2.internal" event={"ID":"b0b12b2fbd9460cffd368b1bb1ef73f5","Type":"ContainerStarted","Data":"85df0a41731d9cc8cf463a5f41e52ca929806cbc0da5de49b0b899e8933b1ad6"}
Sep 08 13:16:11.006384 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.006358    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-178.ec2.internal" event={"ID":"dd5a51648b67abc20f8ee816314baeef","Type":"ContainerStarted","Data":"9533427aa91a2307db0e6df3f4318a0a673be61a86f9b6da2a4eba6b785c481e"}
Sep 08 13:16:11.017974 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.017949    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-systemd-units\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.018110 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.017980    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-etc-sysconfig\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.018110 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018000    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/24bee656-edc4-489d-8955-96306db6aa3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-5ggbg\" (UID: \"24bee656-edc4-489d-8955-96306db6aa3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.018110 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018018    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-cni-binary-copy\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.018110 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018035    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-env-overrides\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.018110 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018064    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-ovnkube-script-lib\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.018110 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018088    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-host-run-netns\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.018110 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018088    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-systemd-units\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.018110 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018113    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjk6k\" (UniqueName: \"kubernetes.io/projected/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-kube-api-access-vjk6k\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.018612 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018139    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bb02bae4-4bf7-4018-b0a3-e9f65b5057fb-hosts-file\") pod \"node-resolver-t8k5r\" (UID: \"bb02bae4-4bf7-4018-b0a3-e9f65b5057fb\") " pod="openshift-dns/node-resolver-t8k5r"
Sep 08 13:16:11.018612 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018186    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzjvz\" (UniqueName: \"kubernetes.io/projected/fca118ca-6b4f-4c1b-9fa7-b384f31387a4-kube-api-access-lzjvz\") pod \"node-ca-f4gjd\" (UID: \"fca118ca-6b4f-4c1b-9fa7-b384f31387a4\") " pod="openshift-image-registry/node-ca-f4gjd"
Sep 08 13:16:11.018612 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018233    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq7mj\" (UniqueName: \"kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj\") pod \"network-check-target-t27j2\" (UID: \"8562e28b-ea18-46bb-98e1-5ea840db0ef5\") " pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:11.018612 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018315    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/987c2846-77e0-499e-9765-0aeb9f290255-iptables-alerter-script\") pod \"iptables-alerter-tjwr7\" (UID: \"987c2846-77e0-499e-9765-0aeb9f290255\") " pod="openshift-network-operator/iptables-alerter-tjwr7"
Sep 08 13:16:11.018612 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018428    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/987c2846-77e0-499e-9765-0aeb9f290255-host-slash\") pod \"iptables-alerter-tjwr7\" (UID: \"987c2846-77e0-499e-9765-0aeb9f290255\") " pod="openshift-network-operator/iptables-alerter-tjwr7"
Sep 08 13:16:11.018612 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018538    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-run-openvswitch\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.018612 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018597    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-run\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.018976 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018624    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-run-openvswitch\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.018976 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018627    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/75d62a9f-0539-4ab9-8f8f-17b833074dfd-etc-tuned\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.018976 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018554    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/987c2846-77e0-499e-9765-0aeb9f290255-host-slash\") pod \"iptables-alerter-tjwr7\" (UID: \"987c2846-77e0-499e-9765-0aeb9f290255\") " pod="openshift-network-operator/iptables-alerter-tjwr7"
Sep 08 13:16:11.018976 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018657    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75d62a9f-0539-4ab9-8f8f-17b833074dfd-tmp\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.018976 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018690    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-env-overrides\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.018976 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018698    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdh9v\" (UniqueName: \"kubernetes.io/projected/246d684e-5ff4-49bb-b39c-660e46385842-kube-api-access-rdh9v\") pod \"network-metrics-daemon-gsgmd\" (UID: \"246d684e-5ff4-49bb-b39c-660e46385842\") " pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:11.018976 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018784    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-ovnkube-script-lib\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.018976 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018795    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-host-kubelet\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.018976 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018838    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-dbus\") pod \"global-pull-secret-syncer-ktjbq\" (UID: \"1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e\") " pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:11.018976 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018901    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-dbus\") pod \"global-pull-secret-syncer-ktjbq\" (UID: \"1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e\") " pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:11.018976 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018915    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-host-kubelet\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.018976 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018930    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/53ffac54-50e1-4c1e-97b1-b18f1164f56d-agent-certs\") pod \"konnectivity-agent-ql8vc\" (UID: \"53ffac54-50e1-4c1e-97b1-b18f1164f56d\") " pod="kube-system/konnectivity-agent-ql8vc"
Sep 08 13:16:11.018976 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.018964    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-etc-sysctl-d\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.019579 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.019494    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fca118ca-6b4f-4c1b-9fa7-b384f31387a4-serviceca\") pod \"node-ca-f4gjd\" (UID: \"fca118ca-6b4f-4c1b-9fa7-b384f31387a4\") " pod="openshift-image-registry/node-ca-f4gjd"
Sep 08 13:16:11.019579 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.019563    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/987c2846-77e0-499e-9765-0aeb9f290255-iptables-alerter-script\") pod \"iptables-alerter-tjwr7\" (UID: \"987c2846-77e0-499e-9765-0aeb9f290255\") " pod="openshift-network-operator/iptables-alerter-tjwr7"
Sep 08 13:16:11.019680 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.019602    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/24bee656-edc4-489d-8955-96306db6aa3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5ggbg\" (UID: \"24bee656-edc4-489d-8955-96306db6aa3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.019680 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.019673    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-multus-conf-dir\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.019777 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.019718    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-log-socket\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.019835 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.019789    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-log-socket\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.019835 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.019790    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-sys\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.019935 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.019838    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-host\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.019935 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.019876    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92c6debe-24d1-449c-bc26-0031c8b9caa1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6f6nt\" (UID: \"92c6debe-24d1-449c-bc26-0031c8b9caa1\") " pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.019935 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.019907    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkmrj\" (UniqueName: \"kubernetes.io/projected/24bee656-edc4-489d-8955-96306db6aa3b-kube-api-access-pkmrj\") pod \"aws-ebs-csi-driver-node-5ggbg\" (UID: \"24bee656-edc4-489d-8955-96306db6aa3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.020082 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.019945    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-etc-kubernetes\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.020082 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.019966    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 13:16:11.020082 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.019986    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g57k\" (UniqueName: \"kubernetes.io/projected/75d62a9f-0539-4ab9-8f8f-17b833074dfd-kube-api-access-8g57k\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.020082 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020049    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92c6debe-24d1-449c-bc26-0031c8b9caa1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6f6nt\" (UID: \"92c6debe-24d1-449c-bc26-0031c8b9caa1\") " pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.020264 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020202    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-host-var-lib-kubelet\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.020264 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020252    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-run-systemd\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.020384 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020287    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-host-cni-bin\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.020384 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92c6debe-24d1-449c-bc26-0031c8b9caa1-cnibin\") pod \"multus-additional-cni-plugins-6f6nt\" (UID: \"92c6debe-24d1-449c-bc26-0031c8b9caa1\") " pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.020384 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020369    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-host-var-lib-cni-bin\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.020544 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020406    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-run-ovn\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.020544 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9k8r\" (UniqueName: \"kubernetes.io/projected/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-kube-api-access-s9k8r\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.020544 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020468    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret\") pod \"global-pull-secret-syncer-ktjbq\" (UID: \"1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e\") " pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:11.020544 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020500    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-cnibin\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.020544 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020535    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-multus-socket-dir-parent\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.020778 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020573    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9z9f\" (UniqueName: \"kubernetes.io/projected/987c2846-77e0-499e-9765-0aeb9f290255-kube-api-access-f9z9f\") pod \"iptables-alerter-tjwr7\" (UID: \"987c2846-77e0-499e-9765-0aeb9f290255\") " pod="openshift-network-operator/iptables-alerter-tjwr7"
Sep 08 13:16:11.020778 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020611    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-host-slash\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.020778 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020645    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-host-run-netns\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.020778 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020687    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-host-run-k8s-cni-cncf-io\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.020778 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020727    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.020778 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020760    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-etc-modprobe-d\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.021050 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020791    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-etc-sysctl-conf\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.021050 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020820    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj9bh\" (UniqueName: \"kubernetes.io/projected/bb02bae4-4bf7-4018-b0a3-e9f65b5057fb-kube-api-access-wj9bh\") pod \"node-resolver-t8k5r\" (UID: \"bb02bae4-4bf7-4018-b0a3-e9f65b5057fb\") " pod="openshift-dns/node-resolver-t8k5r"
Sep 08 13:16:11.021050 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020844    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92c6debe-24d1-449c-bc26-0031c8b9caa1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6f6nt\" (UID: \"92c6debe-24d1-449c-bc26-0031c8b9caa1\") " pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.021050 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020872    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/24bee656-edc4-489d-8955-96306db6aa3b-device-dir\") pod \"aws-ebs-csi-driver-node-5ggbg\" (UID: \"24bee656-edc4-489d-8955-96306db6aa3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.021050 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020900    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-system-cni-dir\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.021050 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.020940    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/24bee656-edc4-489d-8955-96306db6aa3b-sys-fs\") pod \"aws-ebs-csi-driver-node-5ggbg\" (UID: \"24bee656-edc4-489d-8955-96306db6aa3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.021315 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.021136    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-run-systemd\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.021315 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.021155    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-host-slash\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.021315 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.021225    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-host-cni-bin\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.021315 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.021298    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-multus-daemon-config\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.021528 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.021445    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-run-ovn\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.021577 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:11.021530    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:11.021577 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.021553    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.021669 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:11.021606    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret podName:1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e nodeName:}" failed. No retries permitted until 2026-09-08 13:16:11.521584744 +0000 UTC m=+2.982229560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret") pod "global-pull-secret-syncer-ktjbq" (UID: "1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:11.021669 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.021626    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-host-run-netns\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.021779 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.021680    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92c6debe-24d1-449c-bc26-0031c8b9caa1-system-cni-dir\") pod \"multus-additional-cni-plugins-6f6nt\" (UID: \"92c6debe-24d1-449c-bc26-0031c8b9caa1\") " pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.021779 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.021731    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-etc-openvswitch\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.021779 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.021767    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-node-log\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.021958 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.021829    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-host-run-ovn-kubernetes\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.021958 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.021829    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-node-log\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.021958 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.021862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-host-cni-netd\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.021958 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.021924    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-etc-openvswitch\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.021958 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.021943    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-host-run-ovn-kubernetes\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.022198 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.021991    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-ovn-node-metrics-cert\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.022198 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.022049    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-host-cni-netd\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.022198 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.022065    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-kubelet-config\") pod \"global-pull-secret-syncer-ktjbq\" (UID: \"1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e\") " pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:11.022198 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.022189    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fca118ca-6b4f-4c1b-9fa7-b384f31387a4-host\") pod \"node-ca-f4gjd\" (UID: \"fca118ca-6b4f-4c1b-9fa7-b384f31387a4\") " pod="openshift-image-registry/node-ca-f4gjd"
Sep 08 13:16:11.022410 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.022228    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-var-lib-openvswitch\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.022410 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.022261    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/53ffac54-50e1-4c1e-97b1-b18f1164f56d-konnectivity-ca\") pod \"konnectivity-agent-ql8vc\" (UID: \"53ffac54-50e1-4c1e-97b1-b18f1164f56d\") " pod="kube-system/konnectivity-agent-ql8vc"
Sep 08 13:16:11.022410 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.022285    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-kubelet-config\") pod \"global-pull-secret-syncer-ktjbq\" (UID: \"1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e\") " pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:11.022410 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.022319    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-var-lib-kubelet\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.022410 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.022355    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb02bae4-4bf7-4018-b0a3-e9f65b5057fb-tmp-dir\") pod \"node-resolver-t8k5r\" (UID: \"bb02bae4-4bf7-4018-b0a3-e9f65b5057fb\") " pod="openshift-dns/node-resolver-t8k5r"
Sep 08 13:16:11.022410 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.022391    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-var-lib-openvswitch\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.022688 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.022450    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98qps\" (UniqueName: \"kubernetes.io/projected/92c6debe-24d1-449c-bc26-0031c8b9caa1-kube-api-access-98qps\") pod \"multus-additional-cni-plugins-6f6nt\" (UID: \"92c6debe-24d1-449c-bc26-0031c8b9caa1\") " pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.022688 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.022491    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-multus-cni-dir\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.022688 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.022522    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-host-var-lib-cni-multus\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.022688 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.022568    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-host-run-multus-certs\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.022688 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.022604    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs\") pod \"network-metrics-daemon-gsgmd\" (UID: \"246d684e-5ff4-49bb-b39c-660e46385842\") " pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:11.022688 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.022631    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/24bee656-edc4-489d-8955-96306db6aa3b-registration-dir\") pod \"aws-ebs-csi-driver-node-5ggbg\" (UID: \"24bee656-edc4-489d-8955-96306db6aa3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.022688 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.022680    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-hostroot\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.023843 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.022752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-ovnkube-config\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.023843 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.022778    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-etc-systemd\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.023843 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.022806    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-lib-modules\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.023843 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.022838    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92c6debe-24d1-449c-bc26-0031c8b9caa1-cni-binary-copy\") pod \"multus-additional-cni-plugins-6f6nt\" (UID: \"92c6debe-24d1-449c-bc26-0031c8b9caa1\") " pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.023843 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:11.022967    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:11.023843 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.023035    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/53ffac54-50e1-4c1e-97b1-b18f1164f56d-konnectivity-ca\") pod \"konnectivity-agent-ql8vc\" (UID: \"53ffac54-50e1-4c1e-97b1-b18f1164f56d\") " pod="kube-system/konnectivity-agent-ql8vc"
Sep 08 13:16:11.023843 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.022968    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/24bee656-edc4-489d-8955-96306db6aa3b-socket-dir\") pod \"aws-ebs-csi-driver-node-5ggbg\" (UID: \"24bee656-edc4-489d-8955-96306db6aa3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.023843 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:11.023072    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs podName:246d684e-5ff4-49bb-b39c-660e46385842 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:11.523038411 +0000 UTC m=+2.983683216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs") pod "network-metrics-daemon-gsgmd" (UID: "246d684e-5ff4-49bb-b39c-660e46385842") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:11.023843 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.023157    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-etc-kubernetes\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.023843 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.023335    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-ovnkube-config\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.025274 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.025249    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-ovn-node-metrics-cert\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.025610 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.025592    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/53ffac54-50e1-4c1e-97b1-b18f1164f56d-agent-certs\") pod \"konnectivity-agent-ql8vc\" (UID: \"53ffac54-50e1-4c1e-97b1-b18f1164f56d\") " pod="kube-system/konnectivity-agent-ql8vc"
Sep 08 13:16:11.028876 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:11.028857    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:11.028876 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:11.028878    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:11.029015 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:11.028889    2561 projected.go:196] Error preparing data for projected volume kube-api-access-pq7mj for pod openshift-network-diagnostics/network-check-target-t27j2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:11.029015 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:11.028937    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj podName:8562e28b-ea18-46bb-98e1-5ea840db0ef5 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:11.528923093 +0000 UTC m=+2.989567888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pq7mj" (UniqueName: "kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj") pod "network-check-target-t27j2" (UID: "8562e28b-ea18-46bb-98e1-5ea840db0ef5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:11.031841 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.031822    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdh9v\" (UniqueName: \"kubernetes.io/projected/246d684e-5ff4-49bb-b39c-660e46385842-kube-api-access-rdh9v\") pod \"network-metrics-daemon-gsgmd\" (UID: \"246d684e-5ff4-49bb-b39c-660e46385842\") " pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:11.034199 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.034177    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9k8r\" (UniqueName: \"kubernetes.io/projected/fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35-kube-api-access-s9k8r\") pod \"ovnkube-node-mvnln\" (UID: \"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35\") " pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.034360 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.034344    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9z9f\" (UniqueName: \"kubernetes.io/projected/987c2846-77e0-499e-9765-0aeb9f290255-kube-api-access-f9z9f\") pod \"iptables-alerter-tjwr7\" (UID: \"987c2846-77e0-499e-9765-0aeb9f290255\") " pod="openshift-network-operator/iptables-alerter-tjwr7"
Sep 08 13:16:11.123798 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.123763    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-host-run-netns\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.123977 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.123808    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjk6k\" (UniqueName: \"kubernetes.io/projected/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-kube-api-access-vjk6k\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.123977 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.123834    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bb02bae4-4bf7-4018-b0a3-e9f65b5057fb-hosts-file\") pod \"node-resolver-t8k5r\" (UID: \"bb02bae4-4bf7-4018-b0a3-e9f65b5057fb\") " pod="openshift-dns/node-resolver-t8k5r"
Sep 08 13:16:11.123977 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.123860    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzjvz\" (UniqueName: \"kubernetes.io/projected/fca118ca-6b4f-4c1b-9fa7-b384f31387a4-kube-api-access-lzjvz\") pod \"node-ca-f4gjd\" (UID: \"fca118ca-6b4f-4c1b-9fa7-b384f31387a4\") " pod="openshift-image-registry/node-ca-f4gjd"
Sep 08 13:16:11.123977 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.123883    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-host-run-netns\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.123977 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.123899    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-run\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.123977 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.123932    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/75d62a9f-0539-4ab9-8f8f-17b833074dfd-etc-tuned\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.123977 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.123954    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-run\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.123977 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.123959    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75d62a9f-0539-4ab9-8f8f-17b833074dfd-tmp\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.124295 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.123987    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-etc-sysctl-d\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.124295 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.124017    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fca118ca-6b4f-4c1b-9fa7-b384f31387a4-serviceca\") pod \"node-ca-f4gjd\" (UID: \"fca118ca-6b4f-4c1b-9fa7-b384f31387a4\") " pod="openshift-image-registry/node-ca-f4gjd"
Sep 08 13:16:11.124295 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.124024    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bb02bae4-4bf7-4018-b0a3-e9f65b5057fb-hosts-file\") pod \"node-resolver-t8k5r\" (UID: \"bb02bae4-4bf7-4018-b0a3-e9f65b5057fb\") " pod="openshift-dns/node-resolver-t8k5r"
Sep 08 13:16:11.124295 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.124045    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/24bee656-edc4-489d-8955-96306db6aa3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5ggbg\" (UID: \"24bee656-edc4-489d-8955-96306db6aa3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.124295 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.124070    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-multus-conf-dir\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.124295 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.124100    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-sys\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.124295 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.124125    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-host\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.124295 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.124181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-host\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.124295 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.124288    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-multus-conf-dir\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.124738 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.124358    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/24bee656-edc4-489d-8955-96306db6aa3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5ggbg\" (UID: \"24bee656-edc4-489d-8955-96306db6aa3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.124738 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.124490    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-sys\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.124738 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.124595    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-etc-sysctl-d\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.124738 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.124658    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fca118ca-6b4f-4c1b-9fa7-b384f31387a4-serviceca\") pod \"node-ca-f4gjd\" (UID: \"fca118ca-6b4f-4c1b-9fa7-b384f31387a4\") " pod="openshift-image-registry/node-ca-f4gjd"
Sep 08 13:16:11.124738 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.124710    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92c6debe-24d1-449c-bc26-0031c8b9caa1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6f6nt\" (UID: \"92c6debe-24d1-449c-bc26-0031c8b9caa1\") " pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.124964 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.124747    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkmrj\" (UniqueName: \"kubernetes.io/projected/24bee656-edc4-489d-8955-96306db6aa3b-kube-api-access-pkmrj\") pod \"aws-ebs-csi-driver-node-5ggbg\" (UID: \"24bee656-edc4-489d-8955-96306db6aa3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.124964 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.124777    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-etc-kubernetes\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.124964 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.124807    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g57k\" (UniqueName: \"kubernetes.io/projected/75d62a9f-0539-4ab9-8f8f-17b833074dfd-kube-api-access-8g57k\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.124964 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.124833    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92c6debe-24d1-449c-bc26-0031c8b9caa1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6f6nt\" (UID: \"92c6debe-24d1-449c-bc26-0031c8b9caa1\") " pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.124964 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.124860    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-host-var-lib-kubelet\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.124964 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.124906    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92c6debe-24d1-449c-bc26-0031c8b9caa1-cnibin\") pod \"multus-additional-cni-plugins-6f6nt\" (UID: \"92c6debe-24d1-449c-bc26-0031c8b9caa1\") " pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.124964 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.124931    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-host-var-lib-cni-bin\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.125319 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.124970    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-cnibin\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.125319 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125007    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-multus-socket-dir-parent\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.125319 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125053    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-host-run-k8s-cni-cncf-io\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.125319 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125067    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-etc-kubernetes\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.125319 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125082    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-etc-modprobe-d\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.125319 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125106    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-etc-sysctl-conf\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.125319 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125130    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj9bh\" (UniqueName: \"kubernetes.io/projected/bb02bae4-4bf7-4018-b0a3-e9f65b5057fb-kube-api-access-wj9bh\") pod \"node-resolver-t8k5r\" (UID: \"bb02bae4-4bf7-4018-b0a3-e9f65b5057fb\") " pod="openshift-dns/node-resolver-t8k5r"
Sep 08 13:16:11.125319 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125158    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-cnibin\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.125319 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125185    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92c6debe-24d1-449c-bc26-0031c8b9caa1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6f6nt\" (UID: \"92c6debe-24d1-449c-bc26-0031c8b9caa1\") " pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.125319 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125213    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/24bee656-edc4-489d-8955-96306db6aa3b-device-dir\") pod \"aws-ebs-csi-driver-node-5ggbg\" (UID: \"24bee656-edc4-489d-8955-96306db6aa3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.125319 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125237    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92c6debe-24d1-449c-bc26-0031c8b9caa1-cnibin\") pod \"multus-additional-cni-plugins-6f6nt\" (UID: \"92c6debe-24d1-449c-bc26-0031c8b9caa1\") " pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.125319 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125243    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-system-cni-dir\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.125319 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125286    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/24bee656-edc4-489d-8955-96306db6aa3b-sys-fs\") pod \"aws-ebs-csi-driver-node-5ggbg\" (UID: \"24bee656-edc4-489d-8955-96306db6aa3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.125319 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125293    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-system-cni-dir\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.125319 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125316    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-multus-daemon-config\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.125970 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125346    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92c6debe-24d1-449c-bc26-0031c8b9caa1-system-cni-dir\") pod \"multus-additional-cni-plugins-6f6nt\" (UID: \"92c6debe-24d1-449c-bc26-0031c8b9caa1\") " pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.125970 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125369    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-multus-socket-dir-parent\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.125970 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125397    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fca118ca-6b4f-4c1b-9fa7-b384f31387a4-host\") pod \"node-ca-f4gjd\" (UID: \"fca118ca-6b4f-4c1b-9fa7-b384f31387a4\") " pod="openshift-image-registry/node-ca-f4gjd"
Sep 08 13:16:11.125970 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125415    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-host-run-k8s-cni-cncf-io\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.125970 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125427    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-var-lib-kubelet\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.125970 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125453    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb02bae4-4bf7-4018-b0a3-e9f65b5057fb-tmp-dir\") pod \"node-resolver-t8k5r\" (UID: \"bb02bae4-4bf7-4018-b0a3-e9f65b5057fb\") " pod="openshift-dns/node-resolver-t8k5r"
Sep 08 13:16:11.125970 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125481    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98qps\" (UniqueName: \"kubernetes.io/projected/92c6debe-24d1-449c-bc26-0031c8b9caa1-kube-api-access-98qps\") pod \"multus-additional-cni-plugins-6f6nt\" (UID: \"92c6debe-24d1-449c-bc26-0031c8b9caa1\") " pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.125970 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125521    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-etc-modprobe-d\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.125970 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125525    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-multus-cni-dir\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.125970 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125566    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-host-var-lib-cni-multus\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.125970 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125579    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92c6debe-24d1-449c-bc26-0031c8b9caa1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6f6nt\" (UID: \"92c6debe-24d1-449c-bc26-0031c8b9caa1\") " pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.125970 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125592    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-multus-cni-dir\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.125970 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125632    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92c6debe-24d1-449c-bc26-0031c8b9caa1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6f6nt\" (UID: \"92c6debe-24d1-449c-bc26-0031c8b9caa1\") " pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.125970 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125645    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-host-var-lib-cni-bin\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.125970 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125697    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/24bee656-edc4-489d-8955-96306db6aa3b-sys-fs\") pod \"aws-ebs-csi-driver-node-5ggbg\" (UID: \"24bee656-edc4-489d-8955-96306db6aa3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.125970 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125711    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/24bee656-edc4-489d-8955-96306db6aa3b-device-dir\") pod \"aws-ebs-csi-driver-node-5ggbg\" (UID: \"24bee656-edc4-489d-8955-96306db6aa3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.125970 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125764    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92c6debe-24d1-449c-bc26-0031c8b9caa1-system-cni-dir\") pod \"multus-additional-cni-plugins-6f6nt\" (UID: \"92c6debe-24d1-449c-bc26-0031c8b9caa1\") " pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.126574 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125843    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92c6debe-24d1-449c-bc26-0031c8b9caa1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6f6nt\" (UID: \"92c6debe-24d1-449c-bc26-0031c8b9caa1\") " pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.126574 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125872    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-host-var-lib-cni-multus\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.126574 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125873    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-etc-sysctl-conf\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.126574 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125890    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-host-var-lib-kubelet\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.126574 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125942    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-host-run-multus-certs\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.126574 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125953    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-var-lib-kubelet\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.126574 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.125594    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-host-run-multus-certs\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.126574 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126014    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/24bee656-edc4-489d-8955-96306db6aa3b-registration-dir\") pod \"aws-ebs-csi-driver-node-5ggbg\" (UID: \"24bee656-edc4-489d-8955-96306db6aa3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.126574 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126042    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-hostroot\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.126574 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126068    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-etc-systemd\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.126574 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126092    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-lib-modules\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.126574 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126139    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-hostroot\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.126574 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126178    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92c6debe-24d1-449c-bc26-0031c8b9caa1-cni-binary-copy\") pod \"multus-additional-cni-plugins-6f6nt\" (UID: \"92c6debe-24d1-449c-bc26-0031c8b9caa1\") " pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.126574 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126197    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-lib-modules\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.126574 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126209    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb02bae4-4bf7-4018-b0a3-e9f65b5057fb-tmp-dir\") pod \"node-resolver-t8k5r\" (UID: \"bb02bae4-4bf7-4018-b0a3-e9f65b5057fb\") " pod="openshift-dns/node-resolver-t8k5r"
Sep 08 13:16:11.126574 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126230    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/24bee656-edc4-489d-8955-96306db6aa3b-socket-dir\") pod \"aws-ebs-csi-driver-node-5ggbg\" (UID: \"24bee656-edc4-489d-8955-96306db6aa3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.126574 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126267    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-etc-systemd\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.126574 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126280    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-etc-kubernetes\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.127270 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126309    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-etc-sysconfig\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.127270 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126315    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fca118ca-6b4f-4c1b-9fa7-b384f31387a4-host\") pod \"node-ca-f4gjd\" (UID: \"fca118ca-6b4f-4c1b-9fa7-b384f31387a4\") " pod="openshift-image-registry/node-ca-f4gjd"
Sep 08 13:16:11.127270 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126334    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/24bee656-edc4-489d-8955-96306db6aa3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-5ggbg\" (UID: \"24bee656-edc4-489d-8955-96306db6aa3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.127270 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126360    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-cni-binary-copy\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.127270 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126383    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/24bee656-edc4-489d-8955-96306db6aa3b-socket-dir\") pod \"aws-ebs-csi-driver-node-5ggbg\" (UID: \"24bee656-edc4-489d-8955-96306db6aa3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.127270 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126479    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/24bee656-edc4-489d-8955-96306db6aa3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-5ggbg\" (UID: \"24bee656-edc4-489d-8955-96306db6aa3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.127270 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126526    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/24bee656-edc4-489d-8955-96306db6aa3b-registration-dir\") pod \"aws-ebs-csi-driver-node-5ggbg\" (UID: \"24bee656-edc4-489d-8955-96306db6aa3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.127270 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-multus-daemon-config\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.127270 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126602    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/75d62a9f-0539-4ab9-8f8f-17b833074dfd-etc-sysconfig\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.127270 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126640    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-etc-kubernetes\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.127270 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.126920    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-cni-binary-copy\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.127270 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.127012    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92c6debe-24d1-449c-bc26-0031c8b9caa1-cni-binary-copy\") pod \"multus-additional-cni-plugins-6f6nt\" (UID: \"92c6debe-24d1-449c-bc26-0031c8b9caa1\") " pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.127270 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.127106    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/75d62a9f-0539-4ab9-8f8f-17b833074dfd-etc-tuned\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.127270 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.127218    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75d62a9f-0539-4ab9-8f8f-17b833074dfd-tmp\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.134004 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.133900    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzjvz\" (UniqueName: \"kubernetes.io/projected/fca118ca-6b4f-4c1b-9fa7-b384f31387a4-kube-api-access-lzjvz\") pod \"node-ca-f4gjd\" (UID: \"fca118ca-6b4f-4c1b-9fa7-b384f31387a4\") " pod="openshift-image-registry/node-ca-f4gjd"
Sep 08 13:16:11.134004 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.133904    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkmrj\" (UniqueName: \"kubernetes.io/projected/24bee656-edc4-489d-8955-96306db6aa3b-kube-api-access-pkmrj\") pod \"aws-ebs-csi-driver-node-5ggbg\" (UID: \"24bee656-edc4-489d-8955-96306db6aa3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.134292 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.134091    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjk6k\" (UniqueName: \"kubernetes.io/projected/77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa-kube-api-access-vjk6k\") pod \"multus-n4r6n\" (UID: \"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa\") " pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.134966 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.134936    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj9bh\" (UniqueName: \"kubernetes.io/projected/bb02bae4-4bf7-4018-b0a3-e9f65b5057fb-kube-api-access-wj9bh\") pod \"node-resolver-t8k5r\" (UID: \"bb02bae4-4bf7-4018-b0a3-e9f65b5057fb\") " pod="openshift-dns/node-resolver-t8k5r"
Sep 08 13:16:11.136039 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.136014    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98qps\" (UniqueName: \"kubernetes.io/projected/92c6debe-24d1-449c-bc26-0031c8b9caa1-kube-api-access-98qps\") pod \"multus-additional-cni-plugins-6f6nt\" (UID: \"92c6debe-24d1-449c-bc26-0031c8b9caa1\") " pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.136179 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.136150    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g57k\" (UniqueName: \"kubernetes.io/projected/75d62a9f-0539-4ab9-8f8f-17b833074dfd-kube-api-access-8g57k\") pod \"tuned-v25cf\" (UID: \"75d62a9f-0539-4ab9-8f8f-17b833074dfd\") " pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.208891 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.208791    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ql8vc"
Sep 08 13:16:11.218454 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.218424    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tjwr7"
Sep 08 13:16:11.219140 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:11.219108    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ffac54_50e1_4c1e_97b1_b18f1164f56d.slice/crio-fe93cf377113a75593dfac6e50c3407ee7608867222fcf4bc9f11ada55f5495b WatchSource:0}: Error finding container fe93cf377113a75593dfac6e50c3407ee7608867222fcf4bc9f11ada55f5495b: Status 404 returned error can't find the container with id fe93cf377113a75593dfac6e50c3407ee7608867222fcf4bc9f11ada55f5495b
Sep 08 13:16:11.226856 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.226828    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:11.230418 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.230394    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v25cf"
Sep 08 13:16:11.230680 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:11.230653    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987c2846_77e0_499e_9765_0aeb9f290255.slice/crio-80183526c1e550342dabf8f40e13cc39f95fab3099e5dff792f64d484ed67e65 WatchSource:0}: Error finding container 80183526c1e550342dabf8f40e13cc39f95fab3099e5dff792f64d484ed67e65: Status 404 returned error can't find the container with id 80183526c1e550342dabf8f40e13cc39f95fab3099e5dff792f64d484ed67e65
Sep 08 13:16:11.236986 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.236250    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t8k5r"
Sep 08 13:16:11.236986 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:11.236845    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4241b4_6c0f_47bb_8a13_61d4ee7ffd35.slice/crio-8747f544f7e6587e23e9f4bd2500096ec4dab4ea18a612844b6247404fb501f3 WatchSource:0}: Error finding container 8747f544f7e6587e23e9f4bd2500096ec4dab4ea18a612844b6247404fb501f3: Status 404 returned error can't find the container with id 8747f544f7e6587e23e9f4bd2500096ec4dab4ea18a612844b6247404fb501f3
Sep 08 13:16:11.240674 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:11.240646    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d62a9f_0539_4ab9_8f8f_17b833074dfd.slice/crio-080b095ab1784e6a67b8d194e9f9d0ed24dbd8f427481fe94a45b283b97d052f WatchSource:0}: Error finding container 080b095ab1784e6a67b8d194e9f9d0ed24dbd8f427481fe94a45b283b97d052f: Status 404 returned error can't find the container with id 080b095ab1784e6a67b8d194e9f9d0ed24dbd8f427481fe94a45b283b97d052f
Sep 08 13:16:11.241323 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.241283    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6f6nt"
Sep 08 13:16:11.247698 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.247669    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f4gjd"
Sep 08 13:16:11.248115 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:11.248089    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb02bae4_4bf7_4018_b0a3_e9f65b5057fb.slice/crio-b90b472d4941d53f670e6232fe238cba1d356e4583e3ddfc15505ce5e2b52112 WatchSource:0}: Error finding container b90b472d4941d53f670e6232fe238cba1d356e4583e3ddfc15505ce5e2b52112: Status 404 returned error can't find the container with id b90b472d4941d53f670e6232fe238cba1d356e4583e3ddfc15505ce5e2b52112
Sep 08 13:16:11.252005 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:11.251945    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c6debe_24d1_449c_bc26_0031c8b9caa1.slice/crio-36c344db992664eba3e546e5755efdf60b67b76fe2c2c8c396fec690a35b98ed WatchSource:0}: Error finding container 36c344db992664eba3e546e5755efdf60b67b76fe2c2c8c396fec690a35b98ed: Status 404 returned error can't find the container with id 36c344db992664eba3e546e5755efdf60b67b76fe2c2c8c396fec690a35b98ed
Sep 08 13:16:11.253030 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.253009    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg"
Sep 08 13:16:11.257945 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.257920    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n4r6n"
Sep 08 13:16:11.266856 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:11.266827    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca118ca_6b4f_4c1b_9fa7_b384f31387a4.slice/crio-cc4b712f92c6cbd0944e143b8fa4c9ec79d69a558bee103a8a4db8a8015c32ef WatchSource:0}: Error finding container cc4b712f92c6cbd0944e143b8fa4c9ec79d69a558bee103a8a4db8a8015c32ef: Status 404 returned error can't find the container with id cc4b712f92c6cbd0944e143b8fa4c9ec79d69a558bee103a8a4db8a8015c32ef
Sep 08 13:16:11.270865 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:11.270836    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f1e03a_28f3_4b3a_8aad_c2a7f78da7fa.slice/crio-733670d0472488781302c0170525069a09f4f8d793bf4a2e0cd06b157df08760 WatchSource:0}: Error finding container 733670d0472488781302c0170525069a09f4f8d793bf4a2e0cd06b157df08760: Status 404 returned error can't find the container with id 733670d0472488781302c0170525069a09f4f8d793bf4a2e0cd06b157df08760
Sep 08 13:16:11.275444 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:11.274809    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24bee656_edc4_489d_8955_96306db6aa3b.slice/crio-08fb7b9dadd23e9e07cc5361cc786476ccb88f1ffef7e79ffd87ef006f9a7532 WatchSource:0}: Error finding container 08fb7b9dadd23e9e07cc5361cc786476ccb88f1ffef7e79ffd87ef006f9a7532: Status 404 returned error can't find the container with id 08fb7b9dadd23e9e07cc5361cc786476ccb88f1ffef7e79ffd87ef006f9a7532
Sep 08 13:16:11.529615 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.529525    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs\") pod \"network-metrics-daemon-gsgmd\" (UID: \"246d684e-5ff4-49bb-b39c-660e46385842\") " pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:11.529615 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.529574    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq7mj\" (UniqueName: \"kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj\") pod \"network-check-target-t27j2\" (UID: \"8562e28b-ea18-46bb-98e1-5ea840db0ef5\") " pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:11.529864 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.529621    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret\") pod \"global-pull-secret-syncer-ktjbq\" (UID: \"1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e\") " pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:11.529864 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:11.529688    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:11.529864 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:11.529744    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs podName:246d684e-5ff4-49bb-b39c-660e46385842 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:12.529728811 +0000 UTC m=+3.990373604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs") pod "network-metrics-daemon-gsgmd" (UID: "246d684e-5ff4-49bb-b39c-660e46385842") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:11.529864 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:11.529785    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:11.529864 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:11.529809    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:11.529864 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:11.529823    2561 projected.go:196] Error preparing data for projected volume kube-api-access-pq7mj for pod openshift-network-diagnostics/network-check-target-t27j2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:11.530119 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:11.529864    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:11.530119 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:11.529886    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj podName:8562e28b-ea18-46bb-98e1-5ea840db0ef5 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:12.52986872 +0000 UTC m=+3.990513522 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pq7mj" (UniqueName: "kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj") pod "network-check-target-t27j2" (UID: "8562e28b-ea18-46bb-98e1-5ea840db0ef5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:11.530119 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:11.529931    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret podName:1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e nodeName:}" failed. No retries permitted until 2026-09-08 13:16:12.529912491 +0000 UTC m=+3.990557296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret") pod "global-pull-secret-syncer-ktjbq" (UID: "1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:11.962593 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.962468    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 13:11:11 +0000 UTC" deadline="2028-06-03 17:39:47.048622514 +0000 UTC"
Sep 08 13:16:11.962593 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:11.962529    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15220h23m35.086097777s"
Sep 08 13:16:12.015614 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:12.015573    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n4r6n" event={"ID":"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa","Type":"ContainerStarted","Data":"733670d0472488781302c0170525069a09f4f8d793bf4a2e0cd06b157df08760"}
Sep 08 13:16:12.017917 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:12.017882    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg" event={"ID":"24bee656-edc4-489d-8955-96306db6aa3b","Type":"ContainerStarted","Data":"08fb7b9dadd23e9e07cc5361cc786476ccb88f1ffef7e79ffd87ef006f9a7532"}
Sep 08 13:16:12.020793 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:12.020713    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f4gjd" event={"ID":"fca118ca-6b4f-4c1b-9fa7-b384f31387a4","Type":"ContainerStarted","Data":"cc4b712f92c6cbd0944e143b8fa4c9ec79d69a558bee103a8a4db8a8015c32ef"}
Sep 08 13:16:12.022919 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:12.022842    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f6nt" event={"ID":"92c6debe-24d1-449c-bc26-0031c8b9caa1","Type":"ContainerStarted","Data":"36c344db992664eba3e546e5755efdf60b67b76fe2c2c8c396fec690a35b98ed"}
Sep 08 13:16:12.024568 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:12.024505    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v25cf" event={"ID":"75d62a9f-0539-4ab9-8f8f-17b833074dfd","Type":"ContainerStarted","Data":"080b095ab1784e6a67b8d194e9f9d0ed24dbd8f427481fe94a45b283b97d052f"}
Sep 08 13:16:12.027185 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:12.027135    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mvnln" event={"ID":"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35","Type":"ContainerStarted","Data":"8747f544f7e6587e23e9f4bd2500096ec4dab4ea18a612844b6247404fb501f3"}
Sep 08 13:16:12.028883 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:12.028855    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t8k5r" event={"ID":"bb02bae4-4bf7-4018-b0a3-e9f65b5057fb","Type":"ContainerStarted","Data":"b90b472d4941d53f670e6232fe238cba1d356e4583e3ddfc15505ce5e2b52112"}
Sep 08 13:16:12.030568 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:12.030506    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tjwr7" event={"ID":"987c2846-77e0-499e-9765-0aeb9f290255","Type":"ContainerStarted","Data":"80183526c1e550342dabf8f40e13cc39f95fab3099e5dff792f64d484ed67e65"}
Sep 08 13:16:12.032922 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:12.032895    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ql8vc" event={"ID":"53ffac54-50e1-4c1e-97b1-b18f1164f56d","Type":"ContainerStarted","Data":"fe93cf377113a75593dfac6e50c3407ee7608867222fcf4bc9f11ada55f5495b"}
Sep 08 13:16:12.537104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:12.537053    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs\") pod \"network-metrics-daemon-gsgmd\" (UID: \"246d684e-5ff4-49bb-b39c-660e46385842\") " pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:12.537388 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:12.537125    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq7mj\" (UniqueName: \"kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj\") pod \"network-check-target-t27j2\" (UID: \"8562e28b-ea18-46bb-98e1-5ea840db0ef5\") " pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:12.537388 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:12.537172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret\") pod \"global-pull-secret-syncer-ktjbq\" (UID: \"1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e\") " pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:12.537388 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:12.537357    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:12.537584 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:12.537434    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret podName:1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e nodeName:}" failed. No retries permitted until 2026-09-08 13:16:14.537411141 +0000 UTC m=+5.998055946 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret") pod "global-pull-secret-syncer-ktjbq" (UID: "1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:12.537584 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:12.537508    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:12.537584 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:12.537544    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs podName:246d684e-5ff4-49bb-b39c-660e46385842 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:14.537531429 +0000 UTC m=+5.998176235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs") pod "network-metrics-daemon-gsgmd" (UID: "246d684e-5ff4-49bb-b39c-660e46385842") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:12.537797 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:12.537628    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:12.537797 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:12.537643    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:12.537797 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:12.537656    2561 projected.go:196] Error preparing data for projected volume kube-api-access-pq7mj for pod openshift-network-diagnostics/network-check-target-t27j2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:12.537797 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:12.537702    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj podName:8562e28b-ea18-46bb-98e1-5ea840db0ef5 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:14.537679631 +0000 UTC m=+5.998324436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pq7mj" (UniqueName: "kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj") pod "network-check-target-t27j2" (UID: "8562e28b-ea18-46bb-98e1-5ea840db0ef5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:12.962950 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:12.962685    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 13:11:11 +0000 UTC" deadline="2028-02-23 00:26:04.388912968 +0000 UTC"
Sep 08 13:16:12.963409 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:12.962957    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12779h9m51.425966096s"
Sep 08 13:16:13.003345 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:13.003305    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:13.003545 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:13.003471    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktjbq" podUID="1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e"
Sep 08 13:16:13.003623 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:13.003576    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:13.003684 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:13.003664    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gsgmd" podUID="246d684e-5ff4-49bb-b39c-660e46385842"
Sep 08 13:16:13.003737 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:13.003729    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:13.003808 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:13.003791    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t27j2" podUID="8562e28b-ea18-46bb-98e1-5ea840db0ef5"
Sep 08 13:16:14.556646 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:14.555665    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs\") pod \"network-metrics-daemon-gsgmd\" (UID: \"246d684e-5ff4-49bb-b39c-660e46385842\") " pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:14.556646 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:14.555725    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq7mj\" (UniqueName: \"kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj\") pod \"network-check-target-t27j2\" (UID: \"8562e28b-ea18-46bb-98e1-5ea840db0ef5\") " pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:14.556646 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:14.555778    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret\") pod \"global-pull-secret-syncer-ktjbq\" (UID: \"1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e\") " pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:14.556646 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:14.555945    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:14.556646 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:14.556011    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret podName:1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e nodeName:}" failed. No retries permitted until 2026-09-08 13:16:18.555992581 +0000 UTC m=+10.016637377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret") pod "global-pull-secret-syncer-ktjbq" (UID: "1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:14.556646 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:14.556089    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:14.556646 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:14.556127    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs podName:246d684e-5ff4-49bb-b39c-660e46385842 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:18.556114992 +0000 UTC m=+10.016759797 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs") pod "network-metrics-daemon-gsgmd" (UID: "246d684e-5ff4-49bb-b39c-660e46385842") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:14.556646 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:14.556208    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:14.556646 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:14.556222    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:14.556646 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:14.556235    2561 projected.go:196] Error preparing data for projected volume kube-api-access-pq7mj for pod openshift-network-diagnostics/network-check-target-t27j2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:14.556646 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:14.556271    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj podName:8562e28b-ea18-46bb-98e1-5ea840db0ef5 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:18.556260651 +0000 UTC m=+10.016905456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pq7mj" (UniqueName: "kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj") pod "network-check-target-t27j2" (UID: "8562e28b-ea18-46bb-98e1-5ea840db0ef5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:15.003597 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:15.000828    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:15.003597 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:15.000953    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktjbq" podUID="1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e"
Sep 08 13:16:15.003597 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:15.001356    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:15.003597 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:15.001464    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gsgmd" podUID="246d684e-5ff4-49bb-b39c-660e46385842"
Sep 08 13:16:15.003597 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:15.001509    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:15.003597 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:15.001557    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t27j2" podUID="8562e28b-ea18-46bb-98e1-5ea840db0ef5"
Sep 08 13:16:17.001145 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:17.001108    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:17.001597 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:17.001258    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gsgmd" podUID="246d684e-5ff4-49bb-b39c-660e46385842"
Sep 08 13:16:17.001597 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:17.001500    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:17.001597 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:17.001535    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:17.001765 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:17.001607    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t27j2" podUID="8562e28b-ea18-46bb-98e1-5ea840db0ef5"
Sep 08 13:16:17.001765 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:17.001686    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktjbq" podUID="1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e"
Sep 08 13:16:18.590837 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:18.590796    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs\") pod \"network-metrics-daemon-gsgmd\" (UID: \"246d684e-5ff4-49bb-b39c-660e46385842\") " pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:18.591303 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:18.590858    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq7mj\" (UniqueName: \"kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj\") pod \"network-check-target-t27j2\" (UID: \"8562e28b-ea18-46bb-98e1-5ea840db0ef5\") " pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:18.591303 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:18.590902    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret\") pod \"global-pull-secret-syncer-ktjbq\" (UID: \"1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e\") " pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:18.591303 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:18.591029    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:18.591303 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:18.591093    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret podName:1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e nodeName:}" failed. No retries permitted until 2026-09-08 13:16:26.591073991 +0000 UTC m=+18.051718798 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret") pod "global-pull-secret-syncer-ktjbq" (UID: "1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:18.591526 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:18.591492    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:18.591580 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:18.591543    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs podName:246d684e-5ff4-49bb-b39c-660e46385842 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:26.591525845 +0000 UTC m=+18.052170656 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs") pod "network-metrics-daemon-gsgmd" (UID: "246d684e-5ff4-49bb-b39c-660e46385842") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:18.591638 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:18.591619    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:18.591638 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:18.591637    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:18.591730 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:18.591649    2561 projected.go:196] Error preparing data for projected volume kube-api-access-pq7mj for pod openshift-network-diagnostics/network-check-target-t27j2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:18.591730 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:18.591684    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj podName:8562e28b-ea18-46bb-98e1-5ea840db0ef5 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:26.591673192 +0000 UTC m=+18.052317982 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pq7mj" (UniqueName: "kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj") pod "network-check-target-t27j2" (UID: "8562e28b-ea18-46bb-98e1-5ea840db0ef5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:19.003973 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:19.003834    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:19.003973 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:19.003875    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:19.003973 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:19.003844    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:19.003973 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:19.003964    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t27j2" podUID="8562e28b-ea18-46bb-98e1-5ea840db0ef5"
Sep 08 13:16:19.004353 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:19.004056    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktjbq" podUID="1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e"
Sep 08 13:16:19.004353 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:19.004171    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gsgmd" podUID="246d684e-5ff4-49bb-b39c-660e46385842"
Sep 08 13:16:21.005501 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:21.005467    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:21.005965 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:21.005477    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:21.005965 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:21.005604    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktjbq" podUID="1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e"
Sep 08 13:16:21.005965 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:21.005475    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:21.005965 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:21.005669    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t27j2" podUID="8562e28b-ea18-46bb-98e1-5ea840db0ef5"
Sep 08 13:16:21.005965 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:21.005759    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gsgmd" podUID="246d684e-5ff4-49bb-b39c-660e46385842"
Sep 08 13:16:23.000555 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:23.000515    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:23.000555 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:23.000554    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:23.001032 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:23.000515    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:23.001032 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:23.000666    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gsgmd" podUID="246d684e-5ff4-49bb-b39c-660e46385842"
Sep 08 13:16:23.001032 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:23.000793    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t27j2" podUID="8562e28b-ea18-46bb-98e1-5ea840db0ef5"
Sep 08 13:16:23.001032 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:23.000871    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktjbq" podUID="1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e"
Sep 08 13:16:25.001104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:25.001064    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:25.001104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:25.001082    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:25.001625 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:25.001085    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:25.001625 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:25.001193    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t27j2" podUID="8562e28b-ea18-46bb-98e1-5ea840db0ef5"
Sep 08 13:16:25.001625 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:25.001277    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gsgmd" podUID="246d684e-5ff4-49bb-b39c-660e46385842"
Sep 08 13:16:25.001625 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:25.001375    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktjbq" podUID="1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e"
Sep 08 13:16:26.648242 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:26.648208    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret\") pod \"global-pull-secret-syncer-ktjbq\" (UID: \"1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e\") " pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:26.648242 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:26.648256    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs\") pod \"network-metrics-daemon-gsgmd\" (UID: \"246d684e-5ff4-49bb-b39c-660e46385842\") " pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:26.648968 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:26.648297    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq7mj\" (UniqueName: \"kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj\") pod \"network-check-target-t27j2\" (UID: \"8562e28b-ea18-46bb-98e1-5ea840db0ef5\") " pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:26.648968 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:26.648391    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:26.648968 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:26.648402    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:26.648968 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:26.648420    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:26.648968 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:26.648432    2561 projected.go:196] Error preparing data for projected volume kube-api-access-pq7mj for pod openshift-network-diagnostics/network-check-target-t27j2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:26.648968 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:26.648394    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:26.648968 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:26.648461    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs podName:246d684e-5ff4-49bb-b39c-660e46385842 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:42.648441136 +0000 UTC m=+34.109085936 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs") pod "network-metrics-daemon-gsgmd" (UID: "246d684e-5ff4-49bb-b39c-660e46385842") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:26.648968 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:26.648479    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj podName:8562e28b-ea18-46bb-98e1-5ea840db0ef5 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:42.648469553 +0000 UTC m=+34.109114357 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pq7mj" (UniqueName: "kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj") pod "network-check-target-t27j2" (UID: "8562e28b-ea18-46bb-98e1-5ea840db0ef5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:26.648968 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:26.648495    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret podName:1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e nodeName:}" failed. No retries permitted until 2026-09-08 13:16:42.648487794 +0000 UTC m=+34.109132594 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret") pod "global-pull-secret-syncer-ktjbq" (UID: "1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:27.000623 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:27.000585    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:27.000817 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:27.000591    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:27.000817 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:27.000715    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t27j2" podUID="8562e28b-ea18-46bb-98e1-5ea840db0ef5"
Sep 08 13:16:27.000923 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:27.000827    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gsgmd" podUID="246d684e-5ff4-49bb-b39c-660e46385842"
Sep 08 13:16:27.000923 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:27.000593    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:27.001030 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:27.000935    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktjbq" podUID="1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e"
Sep 08 13:16:29.001103 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:29.001066    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:29.001588 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:29.001191    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gsgmd" podUID="246d684e-5ff4-49bb-b39c-660e46385842"
Sep 08 13:16:29.001588 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:29.001239    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:29.001588 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:29.001385    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t27j2" podUID="8562e28b-ea18-46bb-98e1-5ea840db0ef5"
Sep 08 13:16:29.001588 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:29.001435    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:29.001588 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:29.001520    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktjbq" podUID="1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e"
Sep 08 13:16:30.072868 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.072683    2561 generic.go:356] "Generic (PLEG): container finished" podID="dd5a51648b67abc20f8ee816314baeef" containerID="64a3c470d5190297b7e8dad910b88026aab3b0855c3b4b4b88253fa8f1eb8908" exitCode=0
Sep 08 13:16:30.073645 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.072765    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-178.ec2.internal" event={"ID":"dd5a51648b67abc20f8ee816314baeef","Type":"ContainerDied","Data":"64a3c470d5190297b7e8dad910b88026aab3b0855c3b4b4b88253fa8f1eb8908"}
Sep 08 13:16:30.074688 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.074663    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n4r6n" event={"ID":"77f1e03a-28f3-4b3a-8aad-c2a7f78da7fa","Type":"ContainerStarted","Data":"c827a546da705fb4536e819b422c22619987a1c50438d35b6772aa781a188c46"}
Sep 08 13:16:30.076528 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.076424    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg" event={"ID":"24bee656-edc4-489d-8955-96306db6aa3b","Type":"ContainerStarted","Data":"37349d4d21e3253dba839477a30febcf69c40b3cce0d1b47abb215bdf2c8d0d5"}
Sep 08 13:16:30.078095 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.078074    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f4gjd" event={"ID":"fca118ca-6b4f-4c1b-9fa7-b384f31387a4","Type":"ContainerStarted","Data":"7bf9c007ba69d9c2096bf72502254c733cf0a694934ea9f59fc0925a3447d967"}
Sep 08 13:16:30.079756 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.079733    2561 generic.go:356] "Generic (PLEG): container finished" podID="92c6debe-24d1-449c-bc26-0031c8b9caa1" containerID="0defd8d6ddfaf5820da1d12f8ead98f2dcee8dec7d70b96fc7a16bec059306cd" exitCode=0
Sep 08 13:16:30.079877 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.079813    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f6nt" event={"ID":"92c6debe-24d1-449c-bc26-0031c8b9caa1","Type":"ContainerDied","Data":"0defd8d6ddfaf5820da1d12f8ead98f2dcee8dec7d70b96fc7a16bec059306cd"}
Sep 08 13:16:30.081631 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.081606    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v25cf" event={"ID":"75d62a9f-0539-4ab9-8f8f-17b833074dfd","Type":"ContainerStarted","Data":"660062983baaee22da530415201e105320811ff36dc273a526ac24f8e6edccca"}
Sep 08 13:16:30.084954 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.084932    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mvnln" event={"ID":"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35","Type":"ContainerStarted","Data":"8bf4239eb7125f451a4caec4f59350544644e89ba96fb6af6ea4a52d68c34851"}
Sep 08 13:16:30.085047 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.085007    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mvnln" event={"ID":"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35","Type":"ContainerStarted","Data":"5d612751213c72e7123984d07d2a63366d9db24fa1279f638435d2630811cf05"}
Sep 08 13:16:30.085125 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.085060    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mvnln" event={"ID":"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35","Type":"ContainerStarted","Data":"a7ef9892df7d5a450e0471ec38d1ba4c131b86a403684eddbd2ce9fcaf60d692"}
Sep 08 13:16:30.085125 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.085108    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mvnln" event={"ID":"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35","Type":"ContainerStarted","Data":"adf78bb7d47ce6d1ef018bbf6ba2bf4361ea44992cb0be4700342f1eec3a659b"}
Sep 08 13:16:30.085223 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.085154    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mvnln" event={"ID":"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35","Type":"ContainerStarted","Data":"fd3a9c89bb8577a1d26bf547f47067ec8a83f0391e243cea1900a9a131d0c972"}
Sep 08 13:16:30.086779 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.086751    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-178.ec2.internal" event={"ID":"b0b12b2fbd9460cffd368b1bb1ef73f5","Type":"ContainerStarted","Data":"f7a5b8a059718fc2ac67edf34e20f2802acd95b7c309e36ae0179b48ff6f3351"}
Sep 08 13:16:30.088958 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.088938    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t8k5r" event={"ID":"bb02bae4-4bf7-4018-b0a3-e9f65b5057fb","Type":"ContainerStarted","Data":"fd061a89d342580a7e770bb5936ce58ad07f4313af34e5e3a8218db0ba6719ce"}
Sep 08 13:16:30.090467 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.090441    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ql8vc" event={"ID":"53ffac54-50e1-4c1e-97b1-b18f1164f56d","Type":"ContainerStarted","Data":"2beb9346af8c877a4a93395aac9882db8b157f7633ba4df044c1bfe7c90d12da"}
Sep 08 13:16:30.124732 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.124617    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f4gjd" podStartSLOduration=3.031395584 podStartE2EDuration="21.124596925s" totalImagesPullingTime="18.093201341s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:30.124525042 +0000 UTC m=+21.585169853" watchObservedRunningTime="2026-09-08 13:16:30.124596925 +0000 UTC m=+21.585241736"
Sep 08 13:16:30.152486 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.152436    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n4r6n" podStartSLOduration=2.8983665309999997 podStartE2EDuration="21.152418679s" totalImagesPullingTime="18.254052148s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:30.151761896 +0000 UTC m=+21.612406705" watchObservedRunningTime="2026-09-08 13:16:30.152418679 +0000 UTC m=+21.613063490"
Sep 08 13:16:30.161036 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.161007    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ql8vc"
Sep 08 13:16:30.172321 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.172253    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-178.ec2.internal" podStartSLOduration=20.172234713 podStartE2EDuration="20.172234713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:30.171851191 +0000 UTC m=+21.632496002" watchObservedRunningTime="2026-09-08 13:16:30.172234713 +0000 UTC m=+21.632879523"
Sep 08 13:16:30.189430 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.189384    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v25cf" podStartSLOduration=2.90477696 podStartE2EDuration="21.189368506s" totalImagesPullingTime="18.284591546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:30.189099419 +0000 UTC m=+21.649744252" watchObservedRunningTime="2026-09-08 13:16:30.189368506 +0000 UTC m=+21.650013339"
Sep 08 13:16:30.204096 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.204049    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ql8vc" podStartSLOduration=6.8865825449999996 podStartE2EDuration="21.20403631s" totalImagesPullingTime="14.317453765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:30.20392988 +0000 UTC m=+21.664574693" watchObservedRunningTime="2026-09-08 13:16:30.20403631 +0000 UTC m=+21.664681120"
Sep 08 13:16:30.226345 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:30.226281    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-t8k5r" podStartSLOduration=2.949129911 podStartE2EDuration="21.226264839s" totalImagesPullingTime="18.277134928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:30.225651644 +0000 UTC m=+21.686296455" watchObservedRunningTime="2026-09-08 13:16:30.226264839 +0000 UTC m=+21.686909648"
Sep 08 13:16:31.000851 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:31.000763    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:31.001045 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:31.000764    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:31.001045 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:31.000899    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t27j2" podUID="8562e28b-ea18-46bb-98e1-5ea840db0ef5"
Sep 08 13:16:31.001045 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:31.000920    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:31.001045 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:31.000994    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gsgmd" podUID="246d684e-5ff4-49bb-b39c-660e46385842"
Sep 08 13:16:31.001256 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:31.001046    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktjbq" podUID="1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e"
Sep 08 13:16:31.097618 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:31.097576    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mvnln" event={"ID":"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35","Type":"ContainerStarted","Data":"cd5f40bdc1c08d2954e002307a3581a71ebc4d5ce8b350449e7c8cdf07a25a39"}
Sep 08 13:16:31.100369 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:31.099930    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tjwr7" event={"ID":"987c2846-77e0-499e-9765-0aeb9f290255","Type":"ContainerStarted","Data":"e7afe1c88f9ae2c1526f5faa8384cd9621b44f8ba0a269ce30d2b04a3f406a31"}
Sep 08 13:16:31.102383 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:31.102352    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-178.ec2.internal" event={"ID":"dd5a51648b67abc20f8ee816314baeef","Type":"ContainerStarted","Data":"0102f87be953da3d0dd1cd87a82d063302c52d2585e6e57fb8d0297d8e2ce2d2"}
Sep 08 13:16:31.115316 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:31.115195    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tjwr7" podStartSLOduration=3.985693493 podStartE2EDuration="22.11517801s" totalImagesPullingTime="18.129484517s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:31.1144924 +0000 UTC m=+22.575137214" watchObservedRunningTime="2026-09-08 13:16:31.11517801 +0000 UTC m=+22.575822821"
Sep 08 13:16:31.135344 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:31.135274    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-178.ec2.internal" podStartSLOduration=21.135259081 podStartE2EDuration="21.135259081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:31.135184168 +0000 UTC m=+22.595828980" watchObservedRunningTime="2026-09-08 13:16:31.135259081 +0000 UTC m=+22.595903891"
Sep 08 13:16:31.448175 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:31.448150    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 13:16:31.966095 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:31.965744    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T13:16:31.448169983Z","UUID":"d94a24bc-6051-482c-b687-662e5a1c67db","Handler":null,"Name":"","Endpoint":""}
Sep 08 13:16:31.967278 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:31.967258    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 13:16:31.967406 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:31.967310    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 13:16:32.107388 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:32.107350    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg" event={"ID":"24bee656-edc4-489d-8955-96306db6aa3b","Type":"ContainerStarted","Data":"f2b73ce97461ba803d112f3d21a49cc20a7f690096d06166063b4f2c5d7583c3"}
Sep 08 13:16:33.005903 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:33.004323    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:33.005903 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:33.004470    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t27j2" podUID="8562e28b-ea18-46bb-98e1-5ea840db0ef5"
Sep 08 13:16:33.005903 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:33.004543    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:33.005903 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:33.004633    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktjbq" podUID="1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e"
Sep 08 13:16:33.005903 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:33.004699    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:33.005903 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:33.004812    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gsgmd" podUID="246d684e-5ff4-49bb-b39c-660e46385842"
Sep 08 13:16:33.112387 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:33.112336    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg" event={"ID":"24bee656-edc4-489d-8955-96306db6aa3b","Type":"ContainerStarted","Data":"02f0b90e8ef18a4f9480b428f91f0e8ac6e25ca75aab5de5454850622cb36ebc"}
Sep 08 13:16:33.116483 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:33.116454    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mvnln" event={"ID":"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35","Type":"ContainerStarted","Data":"89236b69d913df589dc17fbe0cc7809e8abf5f6ee6a37e31d0ffd38b0aff7f09"}
Sep 08 13:16:33.130502 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:33.130452    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ggbg" podStartSLOduration=2.970417424 podStartE2EDuration="24.130438764s" totalImagesPullingTime="21.16002134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:33.129793432 +0000 UTC m=+24.590438241" watchObservedRunningTime="2026-09-08 13:16:33.130438764 +0000 UTC m=+24.591083573"
Sep 08 13:16:34.483739 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:34.483656    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ql8vc"
Sep 08 13:16:34.484843 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:34.484821    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ql8vc"
Sep 08 13:16:35.000371 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:35.000335    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:35.000541 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:35.000344    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:35.000541 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:35.000428    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t27j2" podUID="8562e28b-ea18-46bb-98e1-5ea840db0ef5"
Sep 08 13:16:35.000652 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:35.000544    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gsgmd" podUID="246d684e-5ff4-49bb-b39c-660e46385842"
Sep 08 13:16:35.000652 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:35.000588    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:35.000759 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:35.000661    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktjbq" podUID="1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e"
Sep 08 13:16:35.125259 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:35.124982    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mvnln" event={"ID":"fa4241b4-6c0f-47bb-8a13-61d4ee7ffd35","Type":"ContainerStarted","Data":"084bc13448b451a1796fdb6778144f9c02d6acef6dc4af0a9add1c01982563e5"}
Sep 08 13:16:35.126138 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:35.125812    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ql8vc"
Sep 08 13:16:35.153940 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:35.153890    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mvnln" podStartSLOduration=7.864463663 podStartE2EDuration="26.153873729s" totalImagesPullingTime="18.289410066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:35.152608946 +0000 UTC m=+26.613253756" watchObservedRunningTime="2026-09-08 13:16:35.153873729 +0000 UTC m=+26.614518540"
Sep 08 13:16:36.129501 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:36.129322    2561 generic.go:356] "Generic (PLEG): container finished" podID="92c6debe-24d1-449c-bc26-0031c8b9caa1" containerID="8463060bba27d89daee616c4c1f523422464039c99e6aca68868bb2ffe5f90c2" exitCode=0
Sep 08 13:16:36.129501 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:36.129408    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f6nt" event={"ID":"92c6debe-24d1-449c-bc26-0031c8b9caa1","Type":"ContainerDied","Data":"8463060bba27d89daee616c4c1f523422464039c99e6aca68868bb2ffe5f90c2"}
Sep 08 13:16:36.130188 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:36.130105    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:36.130188 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:36.130172    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:36.130438 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:36.130214    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:36.145122 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:36.145098    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:36.145248 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:36.145207    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:16:36.986822 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:36.986738    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t27j2"]
Sep 08 13:16:36.986984 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:36.986879    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:36.987039 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:36.987012    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t27j2" podUID="8562e28b-ea18-46bb-98e1-5ea840db0ef5"
Sep 08 13:16:36.987734 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:36.987706    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gsgmd"]
Sep 08 13:16:36.987828 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:36.987814    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:36.987979 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:36.987952    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gsgmd" podUID="246d684e-5ff4-49bb-b39c-660e46385842"
Sep 08 13:16:36.989461 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:36.989436    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ktjbq"]
Sep 08 13:16:36.989553 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:36.989534    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:36.989647 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:36.989626    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktjbq" podUID="1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e"
Sep 08 13:16:37.134247 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:37.134158    2561 generic.go:356] "Generic (PLEG): container finished" podID="92c6debe-24d1-449c-bc26-0031c8b9caa1" containerID="f88530aa8d46d9870c4c3c27c6b7d8285e59169b27be9e81f750497135791ae0" exitCode=0
Sep 08 13:16:37.134247 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:37.134225    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f6nt" event={"ID":"92c6debe-24d1-449c-bc26-0031c8b9caa1","Type":"ContainerDied","Data":"f88530aa8d46d9870c4c3c27c6b7d8285e59169b27be9e81f750497135791ae0"}
Sep 08 13:16:38.138992 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:38.138729    2561 generic.go:356] "Generic (PLEG): container finished" podID="92c6debe-24d1-449c-bc26-0031c8b9caa1" containerID="8a26d2d8012a7af72840590e4e1e17ddbbd0820718dcf7791b063193e8896ac8" exitCode=0
Sep 08 13:16:38.138992 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:38.138816    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f6nt" event={"ID":"92c6debe-24d1-449c-bc26-0031c8b9caa1","Type":"ContainerDied","Data":"8a26d2d8012a7af72840590e4e1e17ddbbd0820718dcf7791b063193e8896ac8"}
Sep 08 13:16:39.001747 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:39.001662    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:39.001747 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:39.001705    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:39.001922 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:39.001794    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:39.001922 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:39.001797    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gsgmd" podUID="246d684e-5ff4-49bb-b39c-660e46385842"
Sep 08 13:16:39.001922 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:39.001875    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktjbq" podUID="1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e"
Sep 08 13:16:39.002022 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:39.001957    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t27j2" podUID="8562e28b-ea18-46bb-98e1-5ea840db0ef5"
Sep 08 13:16:41.000197 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:41.000156    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:41.000197 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:41.000202    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:41.000733 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:41.000156    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:41.000733 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:41.000331    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gsgmd" podUID="246d684e-5ff4-49bb-b39c-660e46385842"
Sep 08 13:16:41.000733 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:41.000415    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t27j2" podUID="8562e28b-ea18-46bb-98e1-5ea840db0ef5"
Sep 08 13:16:41.000733 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:41.000501    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ktjbq" podUID="1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e"
Sep 08 13:16:42.293810 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.293741    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-178.ec2.internal" event="NodeReady"
Sep 08 13:16:42.294180 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.293859    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 13:16:42.346919 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.346880    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-242sh"]
Sep 08 13:16:42.364947 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.364914    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-q5qrk"]
Sep 08 13:16:42.365154 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.365057    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-242sh"
Sep 08 13:16:42.369191 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.369166    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 13:16:42.369348 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.369199    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 13:16:42.369348 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.369267    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8shlb\""
Sep 08 13:16:42.385386 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.385259    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-242sh"]
Sep 08 13:16:42.385386 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.385367    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q5qrk"]
Sep 08 13:16:42.386062 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.385529    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q5qrk"
Sep 08 13:16:42.388125 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.388103    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 13:16:42.388125 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.388119    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8cl4h\""
Sep 08 13:16:42.388298 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.388152    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 13:16:42.388298 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.388239    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 13:16:42.461475 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.461438    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a54b8b62-55df-4e1c-a738-ae5081203413-cert\") pod \"ingress-canary-q5qrk\" (UID: \"a54b8b62-55df-4e1c-a738-ae5081203413\") " pod="openshift-ingress-canary/ingress-canary-q5qrk"
Sep 08 13:16:42.461641 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.461511    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjclb\" (UniqueName: \"kubernetes.io/projected/dcfe8764-419e-438f-91fb-c2f3892df1b2-kube-api-access-qjclb\") pod \"dns-default-242sh\" (UID: \"dcfe8764-419e-438f-91fb-c2f3892df1b2\") " pod="openshift-dns/dns-default-242sh"
Sep 08 13:16:42.461641 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.461538    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5lrc\" (UniqueName: \"kubernetes.io/projected/a54b8b62-55df-4e1c-a738-ae5081203413-kube-api-access-t5lrc\") pod \"ingress-canary-q5qrk\" (UID: \"a54b8b62-55df-4e1c-a738-ae5081203413\") " pod="openshift-ingress-canary/ingress-canary-q5qrk"
Sep 08 13:16:42.461641 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.461570    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls\") pod \"dns-default-242sh\" (UID: \"dcfe8764-419e-438f-91fb-c2f3892df1b2\") " pod="openshift-dns/dns-default-242sh"
Sep 08 13:16:42.461641 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.461600    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfe8764-419e-438f-91fb-c2f3892df1b2-config-volume\") pod \"dns-default-242sh\" (UID: \"dcfe8764-419e-438f-91fb-c2f3892df1b2\") " pod="openshift-dns/dns-default-242sh"
Sep 08 13:16:42.461775 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.461654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcfe8764-419e-438f-91fb-c2f3892df1b2-tmp-dir\") pod \"dns-default-242sh\" (UID: \"dcfe8764-419e-438f-91fb-c2f3892df1b2\") " pod="openshift-dns/dns-default-242sh"
Sep 08 13:16:42.562833 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.562740    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcfe8764-419e-438f-91fb-c2f3892df1b2-tmp-dir\") pod \"dns-default-242sh\" (UID: \"dcfe8764-419e-438f-91fb-c2f3892df1b2\") " pod="openshift-dns/dns-default-242sh"
Sep 08 13:16:42.562833 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.562821    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a54b8b62-55df-4e1c-a738-ae5081203413-cert\") pod \"ingress-canary-q5qrk\" (UID: \"a54b8b62-55df-4e1c-a738-ae5081203413\") " pod="openshift-ingress-canary/ingress-canary-q5qrk"
Sep 08 13:16:42.563071 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.562876    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjclb\" (UniqueName: \"kubernetes.io/projected/dcfe8764-419e-438f-91fb-c2f3892df1b2-kube-api-access-qjclb\") pod \"dns-default-242sh\" (UID: \"dcfe8764-419e-438f-91fb-c2f3892df1b2\") " pod="openshift-dns/dns-default-242sh"
Sep 08 13:16:42.563071 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.562915    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5lrc\" (UniqueName: \"kubernetes.io/projected/a54b8b62-55df-4e1c-a738-ae5081203413-kube-api-access-t5lrc\") pod \"ingress-canary-q5qrk\" (UID: \"a54b8b62-55df-4e1c-a738-ae5081203413\") " pod="openshift-ingress-canary/ingress-canary-q5qrk"
Sep 08 13:16:42.563071 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.562955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls\") pod \"dns-default-242sh\" (UID: \"dcfe8764-419e-438f-91fb-c2f3892df1b2\") " pod="openshift-dns/dns-default-242sh"
Sep 08 13:16:42.563071 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.562981    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfe8764-419e-438f-91fb-c2f3892df1b2-config-volume\") pod \"dns-default-242sh\" (UID: \"dcfe8764-419e-438f-91fb-c2f3892df1b2\") " pod="openshift-dns/dns-default-242sh"
Sep 08 13:16:42.563269 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.563151    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcfe8764-419e-438f-91fb-c2f3892df1b2-tmp-dir\") pod \"dns-default-242sh\" (UID: \"dcfe8764-419e-438f-91fb-c2f3892df1b2\") " pod="openshift-dns/dns-default-242sh"
Sep 08 13:16:42.563269 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:42.563219    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:42.563403 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:42.563283    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls podName:dcfe8764-419e-438f-91fb-c2f3892df1b2 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:43.063263563 +0000 UTC m=+34.523908357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls") pod "dns-default-242sh" (UID: "dcfe8764-419e-438f-91fb-c2f3892df1b2") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:42.563531 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.563513    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfe8764-419e-438f-91fb-c2f3892df1b2-config-volume\") pod \"dns-default-242sh\" (UID: \"dcfe8764-419e-438f-91fb-c2f3892df1b2\") " pod="openshift-dns/dns-default-242sh"
Sep 08 13:16:42.567655 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.567633    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a54b8b62-55df-4e1c-a738-ae5081203413-cert\") pod \"ingress-canary-q5qrk\" (UID: \"a54b8b62-55df-4e1c-a738-ae5081203413\") " pod="openshift-ingress-canary/ingress-canary-q5qrk"
Sep 08 13:16:42.571793 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.571765    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjclb\" (UniqueName: \"kubernetes.io/projected/dcfe8764-419e-438f-91fb-c2f3892df1b2-kube-api-access-qjclb\") pod \"dns-default-242sh\" (UID: \"dcfe8764-419e-438f-91fb-c2f3892df1b2\") " pod="openshift-dns/dns-default-242sh"
Sep 08 13:16:42.572051 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.572029    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5lrc\" (UniqueName: \"kubernetes.io/projected/a54b8b62-55df-4e1c-a738-ae5081203413-kube-api-access-t5lrc\") pod \"ingress-canary-q5qrk\" (UID: \"a54b8b62-55df-4e1c-a738-ae5081203413\") " pod="openshift-ingress-canary/ingress-canary-q5qrk"
Sep 08 13:16:42.663775 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.663735    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs\") pod \"network-metrics-daemon-gsgmd\" (UID: \"246d684e-5ff4-49bb-b39c-660e46385842\") " pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:42.663998 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.663798    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq7mj\" (UniqueName: \"kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj\") pod \"network-check-target-t27j2\" (UID: \"8562e28b-ea18-46bb-98e1-5ea840db0ef5\") " pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:42.663998 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.663840    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret\") pod \"global-pull-secret-syncer-ktjbq\" (UID: \"1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e\") " pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:42.663998 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:42.663910    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:42.663998 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:42.663970    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:42.663998 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:42.663994    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:42.664276 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:42.664016    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:42.664276 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:42.664031    2561 projected.go:196] Error preparing data for projected volume kube-api-access-pq7mj for pod openshift-network-diagnostics/network-check-target-t27j2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:42.664276 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:42.663999    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs podName:246d684e-5ff4-49bb-b39c-660e46385842 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:14.663969971 +0000 UTC m=+66.124614772 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs") pod "network-metrics-daemon-gsgmd" (UID: "246d684e-5ff4-49bb-b39c-660e46385842") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:42.664276 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:42.664073    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret podName:1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e nodeName:}" failed. No retries permitted until 2026-09-08 13:17:14.664046653 +0000 UTC m=+66.124691448 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret") pod "global-pull-secret-syncer-ktjbq" (UID: "1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:42.664276 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:42.664090    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj podName:8562e28b-ea18-46bb-98e1-5ea840db0ef5 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:14.664080964 +0000 UTC m=+66.124725768 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pq7mj" (UniqueName: "kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj") pod "network-check-target-t27j2" (UID: "8562e28b-ea18-46bb-98e1-5ea840db0ef5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:42.695582 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:42.695546    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q5qrk"
Sep 08 13:16:43.003978 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:43.003945    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:16:43.004204 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:43.004073    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:16:43.004204 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:43.004101    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:16:43.007150 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:43.006863    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 13:16:43.007150 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:43.006903    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x5pb6\""
Sep 08 13:16:43.007150 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:43.006904    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4rwvw\""
Sep 08 13:16:43.007150 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:43.006983    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 13:16:43.007150 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:43.006988    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 13:16:43.007150 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:43.006950    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 13:16:43.066840 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:43.066805    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls\") pod \"dns-default-242sh\" (UID: \"dcfe8764-419e-438f-91fb-c2f3892df1b2\") " pod="openshift-dns/dns-default-242sh"
Sep 08 13:16:43.067076 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:43.066970    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:43.067076 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:43.067071    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls podName:dcfe8764-419e-438f-91fb-c2f3892df1b2 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:44.067047358 +0000 UTC m=+35.527692169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls") pod "dns-default-242sh" (UID: "dcfe8764-419e-438f-91fb-c2f3892df1b2") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:44.077031 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:44.076832    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls\") pod \"dns-default-242sh\" (UID: \"dcfe8764-419e-438f-91fb-c2f3892df1b2\") " pod="openshift-dns/dns-default-242sh"
Sep 08 13:16:44.077431 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:44.076977    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:44.077431 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:44.077101    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls podName:dcfe8764-419e-438f-91fb-c2f3892df1b2 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:46.077081418 +0000 UTC m=+37.537726206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls") pod "dns-default-242sh" (UID: "dcfe8764-419e-438f-91fb-c2f3892df1b2") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:44.158257 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:44.158176    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q5qrk"]
Sep 08 13:16:44.177991 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:16:44.177963    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54b8b62_55df_4e1c_a738_ae5081203413.slice/crio-1ea6cf9ed15ab90427d62d44542418f2460cd2b01c5e63118bdb30e097bd06a9 WatchSource:0}: Error finding container 1ea6cf9ed15ab90427d62d44542418f2460cd2b01c5e63118bdb30e097bd06a9: Status 404 returned error can't find the container with id 1ea6cf9ed15ab90427d62d44542418f2460cd2b01c5e63118bdb30e097bd06a9
Sep 08 13:16:45.158417 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:45.158377    2561 generic.go:356] "Generic (PLEG): container finished" podID="92c6debe-24d1-449c-bc26-0031c8b9caa1" containerID="862d92cade225818d99b373b5b4cde8b49b2b10798a9aa14263cf6038b1b8f17" exitCode=0
Sep 08 13:16:45.159078 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:45.158463    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f6nt" event={"ID":"92c6debe-24d1-449c-bc26-0031c8b9caa1","Type":"ContainerDied","Data":"862d92cade225818d99b373b5b4cde8b49b2b10798a9aa14263cf6038b1b8f17"}
Sep 08 13:16:45.159717 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:45.159693    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q5qrk" event={"ID":"a54b8b62-55df-4e1c-a738-ae5081203413","Type":"ContainerStarted","Data":"1ea6cf9ed15ab90427d62d44542418f2460cd2b01c5e63118bdb30e097bd06a9"}
Sep 08 13:16:46.091643 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:46.091586    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls\") pod \"dns-default-242sh\" (UID: \"dcfe8764-419e-438f-91fb-c2f3892df1b2\") " pod="openshift-dns/dns-default-242sh"
Sep 08 13:16:46.091829 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:46.091745    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:46.091829 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:46.091816    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls podName:dcfe8764-419e-438f-91fb-c2f3892df1b2 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:50.091799495 +0000 UTC m=+41.552444283 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls") pod "dns-default-242sh" (UID: "dcfe8764-419e-438f-91fb-c2f3892df1b2") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:46.165553 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:46.165525    2561 generic.go:356] "Generic (PLEG): container finished" podID="92c6debe-24d1-449c-bc26-0031c8b9caa1" containerID="f82b073241bea9ff7cb3c6683f9d64582e38dc21c77897b6b63af7a5b90ab40b" exitCode=0
Sep 08 13:16:46.166121 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:46.165579    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f6nt" event={"ID":"92c6debe-24d1-449c-bc26-0031c8b9caa1","Type":"ContainerDied","Data":"f82b073241bea9ff7cb3c6683f9d64582e38dc21c77897b6b63af7a5b90ab40b"}
Sep 08 13:16:47.171227 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:47.171192    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f6nt" event={"ID":"92c6debe-24d1-449c-bc26-0031c8b9caa1","Type":"ContainerStarted","Data":"f13ee25b64bfc672212c7906aab7c4009ea3715c20e8b8b758f4e5afce4f58cf"}
Sep 08 13:16:47.195676 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:47.195625    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6f6nt" podStartSLOduration=15.021084684 podStartE2EDuration="38.195607048s" totalImagesPullingTime="23.174522364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:47.194925813 +0000 UTC m=+38.655570623" watchObservedRunningTime="2026-09-08 13:16:47.195607048 +0000 UTC m=+38.656251859"
Sep 08 13:16:50.118057 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:50.118009    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls\") pod \"dns-default-242sh\" (UID: \"dcfe8764-419e-438f-91fb-c2f3892df1b2\") " pod="openshift-dns/dns-default-242sh"
Sep 08 13:16:50.118480 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:50.118172    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:50.118480 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:50.118240    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls podName:dcfe8764-419e-438f-91fb-c2f3892df1b2 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:58.118223368 +0000 UTC m=+49.578868156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls") pod "dns-default-242sh" (UID: "dcfe8764-419e-438f-91fb-c2f3892df1b2") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:54.188907 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:54.188872    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q5qrk" event={"ID":"a54b8b62-55df-4e1c-a738-ae5081203413","Type":"ContainerStarted","Data":"5ce2e2a4f3c8fe4b9ddba5cc92c7a9544e8e1f26da6d4bb8445c547491d8920e"}
Sep 08 13:16:54.204902 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:54.204820    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-q5qrk" podStartSLOduration=2.990470721 podStartE2EDuration="12.204805838s" totalImagesPullingTime="9.214335117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:54.204492327 +0000 UTC m=+45.665137140" watchObservedRunningTime="2026-09-08 13:16:54.204805838 +0000 UTC m=+45.665450674"
Sep 08 13:16:58.174131 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:16:58.174087    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls\") pod \"dns-default-242sh\" (UID: \"dcfe8764-419e-438f-91fb-c2f3892df1b2\") " pod="openshift-dns/dns-default-242sh"
Sep 08 13:16:58.174654 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:58.174254    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:58.174654 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:16:58.174347    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls podName:dcfe8764-419e-438f-91fb-c2f3892df1b2 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:14.174329643 +0000 UTC m=+65.634974432 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls") pod "dns-default-242sh" (UID: "dcfe8764-419e-438f-91fb-c2f3892df1b2") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:08.149640 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:08.149608    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mvnln"
Sep 08 13:17:14.186771 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:14.186729    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls\") pod \"dns-default-242sh\" (UID: \"dcfe8764-419e-438f-91fb-c2f3892df1b2\") " pod="openshift-dns/dns-default-242sh"
Sep 08 13:17:14.187151 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:17:14.186873    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:14.187151 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:17:14.186936    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls podName:dcfe8764-419e-438f-91fb-c2f3892df1b2 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:46.186920073 +0000 UTC m=+97.647564861 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls") pod "dns-default-242sh" (UID: "dcfe8764-419e-438f-91fb-c2f3892df1b2") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:14.689969 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:14.689931    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq7mj\" (UniqueName: \"kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj\") pod \"network-check-target-t27j2\" (UID: \"8562e28b-ea18-46bb-98e1-5ea840db0ef5\") " pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:17:14.690145 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:14.689981    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret\") pod \"global-pull-secret-syncer-ktjbq\" (UID: \"1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e\") " pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:17:14.690193 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:14.690141    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs\") pod \"network-metrics-daemon-gsgmd\" (UID: \"246d684e-5ff4-49bb-b39c-660e46385842\") " pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:17:14.692542 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:14.692524    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 13:17:14.692592 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:14.692558    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 13:17:14.692649 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:14.692631    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 13:17:14.700646 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:17:14.700628    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 13:17:14.700698 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:17:14.700679    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs podName:246d684e-5ff4-49bb-b39c-660e46385842 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:18.700664181 +0000 UTC m=+130.161308969 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs") pod "network-metrics-daemon-gsgmd" (UID: "246d684e-5ff4-49bb-b39c-660e46385842") : secret "metrics-daemon-secret" not found
Sep 08 13:17:14.702894 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:14.702872    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 13:17:14.703686 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:14.703669    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e-original-pull-secret\") pod \"global-pull-secret-syncer-ktjbq\" (UID: \"1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e\") " pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:17:14.713448 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:14.713423    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq7mj\" (UniqueName: \"kubernetes.io/projected/8562e28b-ea18-46bb-98e1-5ea840db0ef5-kube-api-access-pq7mj\") pod \"network-check-target-t27j2\" (UID: \"8562e28b-ea18-46bb-98e1-5ea840db0ef5\") " pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:17:14.815384 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:14.815343    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ktjbq"
Sep 08 13:17:14.823814 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:14.823794    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x5pb6\""
Sep 08 13:17:14.831785 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:14.831764    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:17:15.124621 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:15.124518    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ktjbq"]
Sep 08 13:17:15.144054 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:15.144024    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t27j2"]
Sep 08 13:17:15.146962 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:17:15.146938    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8562e28b_ea18_46bb_98e1_5ea840db0ef5.slice/crio-7eab594c8ee56c4fc728a3555ce977fc5c29c36d2799a0acc1c91ae682294a08 WatchSource:0}: Error finding container 7eab594c8ee56c4fc728a3555ce977fc5c29c36d2799a0acc1c91ae682294a08: Status 404 returned error can't find the container with id 7eab594c8ee56c4fc728a3555ce977fc5c29c36d2799a0acc1c91ae682294a08
Sep 08 13:17:15.237267 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:15.237226    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t27j2" event={"ID":"8562e28b-ea18-46bb-98e1-5ea840db0ef5","Type":"ContainerStarted","Data":"7eab594c8ee56c4fc728a3555ce977fc5c29c36d2799a0acc1c91ae682294a08"}
Sep 08 13:17:15.238374 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:15.238348    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ktjbq" event={"ID":"1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e","Type":"ContainerStarted","Data":"97a4d6acb9cae3f3575c6b5ecd41fa11e22eb6f8f69d5d02e10cde29a61a5002"}
Sep 08 13:17:20.254536 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:20.254493    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t27j2" event={"ID":"8562e28b-ea18-46bb-98e1-5ea840db0ef5","Type":"ContainerStarted","Data":"bd86e919030d7d7e781b8060ea154279871546944426d2887169162af6ea8097"}
Sep 08 13:17:20.254999 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:20.254597    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:17:20.256040 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:20.256015    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ktjbq" event={"ID":"1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e","Type":"ContainerStarted","Data":"7fb235a6939df6eeb1fcf98c8f94838807a206f8524918305342ca7bad8b9426"}
Sep 08 13:17:20.272155 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:20.272112    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t27j2" podStartSLOduration=66.754724521 podStartE2EDuration="1m11.272100474s" totalImagesPullingTime="4.517375953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:20.271354538 +0000 UTC m=+71.731999348" watchObservedRunningTime="2026-09-08 13:17:20.272100474 +0000 UTC m=+71.732745283"
Sep 08 13:17:20.286101 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:20.286065    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ktjbq" podStartSLOduration=66.746682103 podStartE2EDuration="1m11.286052142s" totalImagesPullingTime="4.539370039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:20.285853448 +0000 UTC m=+71.746498258" watchObservedRunningTime="2026-09-08 13:17:20.286052142 +0000 UTC m=+71.746696952"
Sep 08 13:17:46.194504 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:46.194364    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls\") pod \"dns-default-242sh\" (UID: \"dcfe8764-419e-438f-91fb-c2f3892df1b2\") " pod="openshift-dns/dns-default-242sh"
Sep 08 13:17:46.194983 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:17:46.194523    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:46.194983 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:17:46.194593    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls podName:dcfe8764-419e-438f-91fb-c2f3892df1b2 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:50.194577214 +0000 UTC m=+161.655222002 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls") pod "dns-default-242sh" (UID: "dcfe8764-419e-438f-91fb-c2f3892df1b2") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:51.261048 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:17:51.261015    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t27j2"
Sep 08 13:18:00.144023 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.143991    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-9q5lg"]
Sep 08 13:18:00.145651 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.145632    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-cvwkr"]
Sep 08 13:18:00.145795 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.145769    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9q5lg"
Sep 08 13:18:00.147929 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.147906    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 08 13:18:00.148168 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.148142    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 08 13:18:00.148945 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.148928    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:18:00.149158 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.149138    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 08 13:18:00.149657 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.149414    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-6g876"]
Sep 08 13:18:00.149657 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.149478    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-nqlkz\""
Sep 08 13:18:00.150128 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.149903    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-cvwkr"
Sep 08 13:18:00.151742 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.151722    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-6g876"
Sep 08 13:18:00.151916 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.151897    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 08 13:18:00.152074 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.152053    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 08 13:18:00.152256 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.152239    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 08 13:18:00.152366 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.152265    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:18:00.152366 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.152265    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-t5mlp\""
Sep 08 13:18:00.153630 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.153612    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-prjzq\""
Sep 08 13:18:00.153776 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.153649    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:18:00.153937 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.153923    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 08 13:18:00.153992 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.153946    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 08 13:18:00.154262 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.154243    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 08 13:18:00.156687 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.156623    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-9q5lg"]
Sep 08 13:18:00.157533 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.157513    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-cvwkr"]
Sep 08 13:18:00.158632 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.158615    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-6g876"]
Sep 08 13:18:00.161997 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.161978    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 08 13:18:00.184641 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.184615    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nvcp\" (UniqueName: \"kubernetes.io/projected/5e21ee11-3e84-4622-8cc7-4984c76637aa-kube-api-access-8nvcp\") pod \"console-operator-85b454874b-6g876\" (UID: \"5e21ee11-3e84-4622-8cc7-4984c76637aa\") " pod="openshift-console-operator/console-operator-85b454874b-6g876"
Sep 08 13:18:00.184743 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.184645    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/509d01ad-7242-4f0f-9897-f0763e1b7298-serving-cert\") pod \"service-ca-operator-859f94c4b-9q5lg\" (UID: \"509d01ad-7242-4f0f-9897-f0763e1b7298\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9q5lg"
Sep 08 13:18:00.184743 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.184687    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/509d01ad-7242-4f0f-9897-f0763e1b7298-config\") pod \"service-ca-operator-859f94c4b-9q5lg\" (UID: \"509d01ad-7242-4f0f-9897-f0763e1b7298\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9q5lg"
Sep 08 13:18:00.184743 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.184713    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/549823a1-8d62-40a7-8de0-62cff329c97d-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-cvwkr\" (UID: \"549823a1-8d62-40a7-8de0-62cff329c97d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-cvwkr"
Sep 08 13:18:00.184743 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.184735    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e21ee11-3e84-4622-8cc7-4984c76637aa-trusted-ca\") pod \"console-operator-85b454874b-6g876\" (UID: \"5e21ee11-3e84-4622-8cc7-4984c76637aa\") " pod="openshift-console-operator/console-operator-85b454874b-6g876"
Sep 08 13:18:00.184883 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.184750    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e21ee11-3e84-4622-8cc7-4984c76637aa-serving-cert\") pod \"console-operator-85b454874b-6g876\" (UID: \"5e21ee11-3e84-4622-8cc7-4984c76637aa\") " pod="openshift-console-operator/console-operator-85b454874b-6g876"
Sep 08 13:18:00.184883 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.184778    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/549823a1-8d62-40a7-8de0-62cff329c97d-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-cvwkr\" (UID: \"549823a1-8d62-40a7-8de0-62cff329c97d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-cvwkr"
Sep 08 13:18:00.184883 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.184840    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvw6s\" (UniqueName: \"kubernetes.io/projected/509d01ad-7242-4f0f-9897-f0763e1b7298-kube-api-access-qvw6s\") pod \"service-ca-operator-859f94c4b-9q5lg\" (UID: \"509d01ad-7242-4f0f-9897-f0763e1b7298\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9q5lg"
Sep 08 13:18:00.184883 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.184872    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e21ee11-3e84-4622-8cc7-4984c76637aa-config\") pod \"console-operator-85b454874b-6g876\" (UID: \"5e21ee11-3e84-4622-8cc7-4984c76637aa\") " pod="openshift-console-operator/console-operator-85b454874b-6g876"
Sep 08 13:18:00.185002 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.184915    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrqvr\" (UniqueName: \"kubernetes.io/projected/549823a1-8d62-40a7-8de0-62cff329c97d-kube-api-access-zrqvr\") pod \"kube-storage-version-migrator-operator-8d7dd4675-cvwkr\" (UID: \"549823a1-8d62-40a7-8de0-62cff329c97d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-cvwkr"
Sep 08 13:18:00.240411 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.240379    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lmphg"]
Sep 08 13:18:00.242131 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.242115    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5457d499bd-hknkx"]
Sep 08 13:18:00.242246 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.242229    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lmphg"
Sep 08 13:18:00.243650 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.243634    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84cdcfc8d-v9vn7"]
Sep 08 13:18:00.243785 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.243765    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.245116 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.245100    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84cdcfc8d-v9vn7"
Sep 08 13:18:00.245910 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.245887    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 08 13:18:00.245997 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.245916    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7424z\""
Sep 08 13:18:00.246159 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.246146    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 13:18:00.246367 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.246348    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:18:00.246835 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.246820    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 13:18:00.248736 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.248671    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 13:18:00.248908 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.248891    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 13:18:00.248977 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.248893    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 13:18:00.249036 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.248999    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 13:18:00.250009 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.249991    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wqhr2\""
Sep 08 13:18:00.250108 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.250018    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-lnnc6\""
Sep 08 13:18:00.250108 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.250019    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 08 13:18:00.253736 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.253650    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lmphg"]
Sep 08 13:18:00.254641 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.254623    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84cdcfc8d-v9vn7"]
Sep 08 13:18:00.255206 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.255188    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 13:18:00.256383 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.256364    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5457d499bd-hknkx"]
Sep 08 13:18:00.285451 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.285421    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nvcp\" (UniqueName: \"kubernetes.io/projected/5e21ee11-3e84-4622-8cc7-4984c76637aa-kube-api-access-8nvcp\") pod \"console-operator-85b454874b-6g876\" (UID: \"5e21ee11-3e84-4622-8cc7-4984c76637aa\") " pod="openshift-console-operator/console-operator-85b454874b-6g876"
Sep 08 13:18:00.285659 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.285637    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-tls\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.285771 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.285670    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac87abf4-bf58-4f6b-897c-ed20252c6b33-installation-pull-secrets\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.285771 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.285697    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5pxd\" (UniqueName: \"kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-kube-api-access-g5pxd\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.285771 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.285721    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfcs5\" (UniqueName: \"kubernetes.io/projected/8e733f5a-3d38-4093-8f32-3fec4138aa41-kube-api-access-gfcs5\") pod \"managed-serviceaccount-addon-agent-84cdcfc8d-v9vn7\" (UID: \"8e733f5a-3d38-4093-8f32-3fec4138aa41\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84cdcfc8d-v9vn7"
Sep 08 13:18:00.285881 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.285783    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/509d01ad-7242-4f0f-9897-f0763e1b7298-serving-cert\") pod \"service-ca-operator-859f94c4b-9q5lg\" (UID: \"509d01ad-7242-4f0f-9897-f0763e1b7298\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9q5lg"
Sep 08 13:18:00.285881 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.285808    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/509d01ad-7242-4f0f-9897-f0763e1b7298-config\") pod \"service-ca-operator-859f94c4b-9q5lg\" (UID: \"509d01ad-7242-4f0f-9897-f0763e1b7298\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9q5lg"
Sep 08 13:18:00.285881 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.285831    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/549823a1-8d62-40a7-8de0-62cff329c97d-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-cvwkr\" (UID: \"549823a1-8d62-40a7-8de0-62cff329c97d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-cvwkr"
Sep 08 13:18:00.285881 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.285859    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e21ee11-3e84-4622-8cc7-4984c76637aa-trusted-ca\") pod \"console-operator-85b454874b-6g876\" (UID: \"5e21ee11-3e84-4622-8cc7-4984c76637aa\") " pod="openshift-console-operator/console-operator-85b454874b-6g876"
Sep 08 13:18:00.286051 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.285883    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e21ee11-3e84-4622-8cc7-4984c76637aa-serving-cert\") pod \"console-operator-85b454874b-6g876\" (UID: \"5e21ee11-3e84-4622-8cc7-4984c76637aa\") " pod="openshift-console-operator/console-operator-85b454874b-6g876"
Sep 08 13:18:00.286051 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.285908    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac87abf4-bf58-4f6b-897c-ed20252c6b33-image-registry-private-configuration\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.286051 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.285931    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac87abf4-bf58-4f6b-897c-ed20252c6b33-ca-trust-extracted\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.286051 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.285956    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-certificates\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.286051 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.285977    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac87abf4-bf58-4f6b-897c-ed20252c6b33-trusted-ca\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.286051 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.286006    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/549823a1-8d62-40a7-8de0-62cff329c97d-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-cvwkr\" (UID: \"549823a1-8d62-40a7-8de0-62cff329c97d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-cvwkr"
Sep 08 13:18:00.286051 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.286042    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvw6s\" (UniqueName: \"kubernetes.io/projected/509d01ad-7242-4f0f-9897-f0763e1b7298-kube-api-access-qvw6s\") pod \"service-ca-operator-859f94c4b-9q5lg\" (UID: \"509d01ad-7242-4f0f-9897-f0763e1b7298\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9q5lg"
Sep 08 13:18:00.286427 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.286070    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ht7k\" (UniqueName: \"kubernetes.io/projected/bc3bc449-a85e-4f82-bede-62b391eac326-kube-api-access-7ht7k\") pod \"volume-data-source-validator-695c895bd6-lmphg\" (UID: \"bc3bc449-a85e-4f82-bede-62b391eac326\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lmphg"
Sep 08 13:18:00.286427 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.286130    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e21ee11-3e84-4622-8cc7-4984c76637aa-config\") pod \"console-operator-85b454874b-6g876\" (UID: \"5e21ee11-3e84-4622-8cc7-4984c76637aa\") " pod="openshift-console-operator/console-operator-85b454874b-6g876"
Sep 08 13:18:00.286427 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.286187    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e733f5a-3d38-4093-8f32-3fec4138aa41-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-84cdcfc8d-v9vn7\" (UID: \"8e733f5a-3d38-4093-8f32-3fec4138aa41\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84cdcfc8d-v9vn7"
Sep 08 13:18:00.286427 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.286215    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-bound-sa-token\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.286427 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.286255    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrqvr\" (UniqueName: \"kubernetes.io/projected/549823a1-8d62-40a7-8de0-62cff329c97d-kube-api-access-zrqvr\") pod \"kube-storage-version-migrator-operator-8d7dd4675-cvwkr\" (UID: \"549823a1-8d62-40a7-8de0-62cff329c97d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-cvwkr"
Sep 08 13:18:00.286675 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.286562    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/549823a1-8d62-40a7-8de0-62cff329c97d-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-cvwkr\" (UID: \"549823a1-8d62-40a7-8de0-62cff329c97d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-cvwkr"
Sep 08 13:18:00.286675 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.286565    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/509d01ad-7242-4f0f-9897-f0763e1b7298-config\") pod \"service-ca-operator-859f94c4b-9q5lg\" (UID: \"509d01ad-7242-4f0f-9897-f0763e1b7298\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9q5lg"
Sep 08 13:18:00.287160 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.287133    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e21ee11-3e84-4622-8cc7-4984c76637aa-config\") pod \"console-operator-85b454874b-6g876\" (UID: \"5e21ee11-3e84-4622-8cc7-4984c76637aa\") " pod="openshift-console-operator/console-operator-85b454874b-6g876"
Sep 08 13:18:00.287368 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.287350    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e21ee11-3e84-4622-8cc7-4984c76637aa-trusted-ca\") pod \"console-operator-85b454874b-6g876\" (UID: \"5e21ee11-3e84-4622-8cc7-4984c76637aa\") " pod="openshift-console-operator/console-operator-85b454874b-6g876"
Sep 08 13:18:00.288163 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.288139    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/509d01ad-7242-4f0f-9897-f0763e1b7298-serving-cert\") pod \"service-ca-operator-859f94c4b-9q5lg\" (UID: \"509d01ad-7242-4f0f-9897-f0763e1b7298\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9q5lg"
Sep 08 13:18:00.288274 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.288257    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e21ee11-3e84-4622-8cc7-4984c76637aa-serving-cert\") pod \"console-operator-85b454874b-6g876\" (UID: \"5e21ee11-3e84-4622-8cc7-4984c76637aa\") " pod="openshift-console-operator/console-operator-85b454874b-6g876"
Sep 08 13:18:00.288596 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.288572    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/549823a1-8d62-40a7-8de0-62cff329c97d-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-cvwkr\" (UID: \"549823a1-8d62-40a7-8de0-62cff329c97d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-cvwkr"
Sep 08 13:18:00.303967 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.303937    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvw6s\" (UniqueName: \"kubernetes.io/projected/509d01ad-7242-4f0f-9897-f0763e1b7298-kube-api-access-qvw6s\") pod \"service-ca-operator-859f94c4b-9q5lg\" (UID: \"509d01ad-7242-4f0f-9897-f0763e1b7298\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9q5lg"
Sep 08 13:18:00.304135 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.304119    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nvcp\" (UniqueName: \"kubernetes.io/projected/5e21ee11-3e84-4622-8cc7-4984c76637aa-kube-api-access-8nvcp\") pod \"console-operator-85b454874b-6g876\" (UID: \"5e21ee11-3e84-4622-8cc7-4984c76637aa\") " pod="openshift-console-operator/console-operator-85b454874b-6g876"
Sep 08 13:18:00.304761 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.304746    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrqvr\" (UniqueName: \"kubernetes.io/projected/549823a1-8d62-40a7-8de0-62cff329c97d-kube-api-access-zrqvr\") pod \"kube-storage-version-migrator-operator-8d7dd4675-cvwkr\" (UID: \"549823a1-8d62-40a7-8de0-62cff329c97d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-cvwkr"
Sep 08 13:18:00.336901 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.336875    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl"]
Sep 08 13:18:00.339553 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.339528    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl"
Sep 08 13:18:00.341917 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.341899    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 13:18:00.342015 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.341963    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-m5dl8\""
Sep 08 13:18:00.342074 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.342017    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 08 13:18:00.342074 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.342024    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 08 13:18:00.342430 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.342411    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 13:18:00.350396 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.350373    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl"]
Sep 08 13:18:00.386834 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.386804    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5pxd\" (UniqueName: \"kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-kube-api-access-g5pxd\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.386834 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.386834    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfcs5\" (UniqueName: \"kubernetes.io/projected/8e733f5a-3d38-4093-8f32-3fec4138aa41-kube-api-access-gfcs5\") pod \"managed-serviceaccount-addon-agent-84cdcfc8d-v9vn7\" (UID: \"8e733f5a-3d38-4093-8f32-3fec4138aa41\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84cdcfc8d-v9vn7"
Sep 08 13:18:00.387006 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.386863    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac87abf4-bf58-4f6b-897c-ed20252c6b33-ca-trust-extracted\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.387006 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.386886    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac87abf4-bf58-4f6b-897c-ed20252c6b33-trusted-ca\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.387006 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.386917    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ht7k\" (UniqueName: \"kubernetes.io/projected/bc3bc449-a85e-4f82-bede-62b391eac326-kube-api-access-7ht7k\") pod \"volume-data-source-validator-695c895bd6-lmphg\" (UID: \"bc3bc449-a85e-4f82-bede-62b391eac326\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lmphg"
Sep 08 13:18:00.387006 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.386942    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e733f5a-3d38-4093-8f32-3fec4138aa41-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-84cdcfc8d-v9vn7\" (UID: \"8e733f5a-3d38-4093-8f32-3fec4138aa41\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84cdcfc8d-v9vn7"
Sep 08 13:18:00.387006 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.386968    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-bound-sa-token\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.387006 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.386994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac87abf4-bf58-4f6b-897c-ed20252c6b33-image-registry-private-configuration\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.387267 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.387022    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac87abf4-bf58-4f6b-897c-ed20252c6b33-installation-pull-secrets\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.387267 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.387045    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-tls\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.387267 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.387085    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/773cfcaf-f677-4056-bbe8-1a475443aa1b-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-dj7cl\" (UID: \"773cfcaf-f677-4056-bbe8-1a475443aa1b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl"
Sep 08 13:18:00.387267 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.387111    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmxcw\" (UniqueName: \"kubernetes.io/projected/773cfcaf-f677-4056-bbe8-1a475443aa1b-kube-api-access-jmxcw\") pod \"cluster-monitoring-operator-6cdcc96965-dj7cl\" (UID: \"773cfcaf-f677-4056-bbe8-1a475443aa1b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl"
Sep 08 13:18:00.387489 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.387273    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/773cfcaf-f677-4056-bbe8-1a475443aa1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dj7cl\" (UID: \"773cfcaf-f677-4056-bbe8-1a475443aa1b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl"
Sep 08 13:18:00.387489 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.387280    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac87abf4-bf58-4f6b-897c-ed20252c6b33-ca-trust-extracted\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.387489 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:00.387382    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:18:00.387489 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:00.387397    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5457d499bd-hknkx: secret "image-registry-tls" not found
Sep 08 13:18:00.387489 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.387434    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-certificates\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.387489 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:00.387463    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-tls podName:ac87abf4-bf58-4f6b-897c-ed20252c6b33 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:00.887443549 +0000 UTC m=+112.348088347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-tls") pod "image-registry-5457d499bd-hknkx" (UID: "ac87abf4-bf58-4f6b-897c-ed20252c6b33") : secret "image-registry-tls" not found
Sep 08 13:18:00.387995 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.387970    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-certificates\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.388074 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.387979    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac87abf4-bf58-4f6b-897c-ed20252c6b33-trusted-ca\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.389620 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.389588    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac87abf4-bf58-4f6b-897c-ed20252c6b33-installation-pull-secrets\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.389620 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.389607    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e733f5a-3d38-4093-8f32-3fec4138aa41-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-84cdcfc8d-v9vn7\" (UID: \"8e733f5a-3d38-4093-8f32-3fec4138aa41\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84cdcfc8d-v9vn7"
Sep 08 13:18:00.389732 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.389588    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac87abf4-bf58-4f6b-897c-ed20252c6b33-image-registry-private-configuration\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.395753 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.395696    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-bound-sa-token\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.396089 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.396063    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfcs5\" (UniqueName: \"kubernetes.io/projected/8e733f5a-3d38-4093-8f32-3fec4138aa41-kube-api-access-gfcs5\") pod \"managed-serviceaccount-addon-agent-84cdcfc8d-v9vn7\" (UID: \"8e733f5a-3d38-4093-8f32-3fec4138aa41\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84cdcfc8d-v9vn7"
Sep 08 13:18:00.396504 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.396488    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5pxd\" (UniqueName: \"kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-kube-api-access-g5pxd\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.396746 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.396730    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ht7k\" (UniqueName: \"kubernetes.io/projected/bc3bc449-a85e-4f82-bede-62b391eac326-kube-api-access-7ht7k\") pod \"volume-data-source-validator-695c895bd6-lmphg\" (UID: \"bc3bc449-a85e-4f82-bede-62b391eac326\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lmphg"
Sep 08 13:18:00.458755 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.458723    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9q5lg"
Sep 08 13:18:00.467378 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.467353    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-cvwkr"
Sep 08 13:18:00.471389 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.471366    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-6g876"
Sep 08 13:18:00.488669 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.488636    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/773cfcaf-f677-4056-bbe8-1a475443aa1b-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-dj7cl\" (UID: \"773cfcaf-f677-4056-bbe8-1a475443aa1b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl"
Sep 08 13:18:00.488770 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.488683    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmxcw\" (UniqueName: \"kubernetes.io/projected/773cfcaf-f677-4056-bbe8-1a475443aa1b-kube-api-access-jmxcw\") pod \"cluster-monitoring-operator-6cdcc96965-dj7cl\" (UID: \"773cfcaf-f677-4056-bbe8-1a475443aa1b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl"
Sep 08 13:18:00.488867 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.488827    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/773cfcaf-f677-4056-bbe8-1a475443aa1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dj7cl\" (UID: \"773cfcaf-f677-4056-bbe8-1a475443aa1b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl"
Sep 08 13:18:00.489097 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:00.489059    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 13:18:00.489213 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:00.489163    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/773cfcaf-f677-4056-bbe8-1a475443aa1b-cluster-monitoring-operator-tls podName:773cfcaf-f677-4056-bbe8-1a475443aa1b nodeName:}" failed. No retries permitted until 2026-09-08 13:18:00.989139783 +0000 UTC m=+112.449784617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/773cfcaf-f677-4056-bbe8-1a475443aa1b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dj7cl" (UID: "773cfcaf-f677-4056-bbe8-1a475443aa1b") : secret "cluster-monitoring-operator-tls" not found
Sep 08 13:18:00.489443 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.489419    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/773cfcaf-f677-4056-bbe8-1a475443aa1b-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-dj7cl\" (UID: \"773cfcaf-f677-4056-bbe8-1a475443aa1b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl"
Sep 08 13:18:00.499098 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.499064    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmxcw\" (UniqueName: \"kubernetes.io/projected/773cfcaf-f677-4056-bbe8-1a475443aa1b-kube-api-access-jmxcw\") pod \"cluster-monitoring-operator-6cdcc96965-dj7cl\" (UID: \"773cfcaf-f677-4056-bbe8-1a475443aa1b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl"
Sep 08 13:18:00.552471 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.552441    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lmphg"
Sep 08 13:18:00.575734 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.575627    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84cdcfc8d-v9vn7"
Sep 08 13:18:00.732047 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.732011    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-9q5lg"]
Sep 08 13:18:00.735185 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.735157    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-cvwkr"]
Sep 08 13:18:00.735340 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:18:00.735238    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509d01ad_7242_4f0f_9897_f0763e1b7298.slice/crio-1b0a8411913b503addc05e72d3b3f34cbda6a28fe6bf2960cd1ea90b0d8abeb6 WatchSource:0}: Error finding container 1b0a8411913b503addc05e72d3b3f34cbda6a28fe6bf2960cd1ea90b0d8abeb6: Status 404 returned error can't find the container with id 1b0a8411913b503addc05e72d3b3f34cbda6a28fe6bf2960cd1ea90b0d8abeb6
Sep 08 13:18:00.737650 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:18:00.737626    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549823a1_8d62_40a7_8de0_62cff329c97d.slice/crio-a8ac2c07c6bab94899aed929e8de4572239a75146e4a0940b0af01913ada0549 WatchSource:0}: Error finding container a8ac2c07c6bab94899aed929e8de4572239a75146e4a0940b0af01913ada0549: Status 404 returned error can't find the container with id a8ac2c07c6bab94899aed929e8de4572239a75146e4a0940b0af01913ada0549
Sep 08 13:18:00.761753 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.761721    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-6g876"]
Sep 08 13:18:00.764203 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:18:00.764177    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e21ee11_3e84_4622_8cc7_4984c76637aa.slice/crio-8df84059e88767ac24a61aff131afc367ce0f01d7a5ff220d634c62a6aa7d36f WatchSource:0}: Error finding container 8df84059e88767ac24a61aff131afc367ce0f01d7a5ff220d634c62a6aa7d36f: Status 404 returned error can't find the container with id 8df84059e88767ac24a61aff131afc367ce0f01d7a5ff220d634c62a6aa7d36f
Sep 08 13:18:00.823432 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.823404    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lmphg"]
Sep 08 13:18:00.826220 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:18:00.826196    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3bc449_a85e_4f82_bede_62b391eac326.slice/crio-845651a6ff825915c6962ed7d07a7b1be4c3e18cd770920fa378f24b92bf7dbe WatchSource:0}: Error finding container 845651a6ff825915c6962ed7d07a7b1be4c3e18cd770920fa378f24b92bf7dbe: Status 404 returned error can't find the container with id 845651a6ff825915c6962ed7d07a7b1be4c3e18cd770920fa378f24b92bf7dbe
Sep 08 13:18:00.853302 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.853261    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84cdcfc8d-v9vn7"]
Sep 08 13:18:00.856347 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:18:00.856324    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e733f5a_3d38_4093_8f32_3fec4138aa41.slice/crio-b9d1c5ed8821f40dce029949753fb8f8e2aa2876c2a6cdf1a5d169a1f83396e3 WatchSource:0}: Error finding container b9d1c5ed8821f40dce029949753fb8f8e2aa2876c2a6cdf1a5d169a1f83396e3: Status 404 returned error can't find the container with id b9d1c5ed8821f40dce029949753fb8f8e2aa2876c2a6cdf1a5d169a1f83396e3
Sep 08 13:18:00.892552 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.892522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-tls\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:00.892692 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:00.892674    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:18:00.892749 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:00.892694    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5457d499bd-hknkx: secret "image-registry-tls" not found
Sep 08 13:18:00.892749 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:00.892744    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-tls podName:ac87abf4-bf58-4f6b-897c-ed20252c6b33 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:01.89273021 +0000 UTC m=+113.353374998 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-tls") pod "image-registry-5457d499bd-hknkx" (UID: "ac87abf4-bf58-4f6b-897c-ed20252c6b33") : secret "image-registry-tls" not found
Sep 08 13:18:00.993957 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:00.993866    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/773cfcaf-f677-4056-bbe8-1a475443aa1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dj7cl\" (UID: \"773cfcaf-f677-4056-bbe8-1a475443aa1b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl"
Sep 08 13:18:00.994114 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:00.993982    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 13:18:00.994114 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:00.994041    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/773cfcaf-f677-4056-bbe8-1a475443aa1b-cluster-monitoring-operator-tls podName:773cfcaf-f677-4056-bbe8-1a475443aa1b nodeName:}" failed. No retries permitted until 2026-09-08 13:18:01.99402427 +0000 UTC m=+113.454669060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/773cfcaf-f677-4056-bbe8-1a475443aa1b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dj7cl" (UID: "773cfcaf-f677-4056-bbe8-1a475443aa1b") : secret "cluster-monitoring-operator-tls" not found
Sep 08 13:18:01.353752 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:01.353657    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lmphg" event={"ID":"bc3bc449-a85e-4f82-bede-62b391eac326","Type":"ContainerStarted","Data":"845651a6ff825915c6962ed7d07a7b1be4c3e18cd770920fa378f24b92bf7dbe"}
Sep 08 13:18:01.354853 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:01.354830    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-cvwkr" event={"ID":"549823a1-8d62-40a7-8de0-62cff329c97d","Type":"ContainerStarted","Data":"a8ac2c07c6bab94899aed929e8de4572239a75146e4a0940b0af01913ada0549"}
Sep 08 13:18:01.355848 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:01.355826    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9q5lg" event={"ID":"509d01ad-7242-4f0f-9897-f0763e1b7298","Type":"ContainerStarted","Data":"1b0a8411913b503addc05e72d3b3f34cbda6a28fe6bf2960cd1ea90b0d8abeb6"}
Sep 08 13:18:01.356927 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:01.356903    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-6g876" event={"ID":"5e21ee11-3e84-4622-8cc7-4984c76637aa","Type":"ContainerStarted","Data":"8df84059e88767ac24a61aff131afc367ce0f01d7a5ff220d634c62a6aa7d36f"}
Sep 08 13:18:01.357935 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:01.357912    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84cdcfc8d-v9vn7" event={"ID":"8e733f5a-3d38-4093-8f32-3fec4138aa41","Type":"ContainerStarted","Data":"b9d1c5ed8821f40dce029949753fb8f8e2aa2876c2a6cdf1a5d169a1f83396e3"}
Sep 08 13:18:01.902261 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:01.902217    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-tls\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:01.902508 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:01.902488    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:18:01.902508 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:01.902509    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5457d499bd-hknkx: secret "image-registry-tls" not found
Sep 08 13:18:01.902649 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:01.902569    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-tls podName:ac87abf4-bf58-4f6b-897c-ed20252c6b33 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:03.902549453 +0000 UTC m=+115.363194246 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-tls") pod "image-registry-5457d499bd-hknkx" (UID: "ac87abf4-bf58-4f6b-897c-ed20252c6b33") : secret "image-registry-tls" not found
Sep 08 13:18:02.003268 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:02.003224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/773cfcaf-f677-4056-bbe8-1a475443aa1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dj7cl\" (UID: \"773cfcaf-f677-4056-bbe8-1a475443aa1b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl"
Sep 08 13:18:02.003478 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:02.003438    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 13:18:02.003543 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:02.003514    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/773cfcaf-f677-4056-bbe8-1a475443aa1b-cluster-monitoring-operator-tls podName:773cfcaf-f677-4056-bbe8-1a475443aa1b nodeName:}" failed. No retries permitted until 2026-09-08 13:18:04.003494725 +0000 UTC m=+115.464139515 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/773cfcaf-f677-4056-bbe8-1a475443aa1b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dj7cl" (UID: "773cfcaf-f677-4056-bbe8-1a475443aa1b") : secret "cluster-monitoring-operator-tls" not found
Sep 08 13:18:03.364695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:03.364653    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lmphg" event={"ID":"bc3bc449-a85e-4f82-bede-62b391eac326","Type":"ContainerStarted","Data":"db41d15fcc59c970d9369f96472f586cfc45102c068aa14102203abd0d030347"}
Sep 08 13:18:03.380979 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:03.380927    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lmphg" podStartSLOduration=1.794995196 podStartE2EDuration="3.380912283s" totalImagesPullingTime="1.585917087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:03.38048214 +0000 UTC m=+114.841126951" watchObservedRunningTime="2026-09-08 13:18:03.380912283 +0000 UTC m=+114.841557083"
Sep 08 13:18:03.919735 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:03.919697    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-tls\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:03.919907 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:03.919822    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:18:03.919907 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:03.919834    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5457d499bd-hknkx: secret "image-registry-tls" not found
Sep 08 13:18:03.919907 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:03.919905    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-tls podName:ac87abf4-bf58-4f6b-897c-ed20252c6b33 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:07.919886114 +0000 UTC m=+119.380530904 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-tls") pod "image-registry-5457d499bd-hknkx" (UID: "ac87abf4-bf58-4f6b-897c-ed20252c6b33") : secret "image-registry-tls" not found
Sep 08 13:18:04.020057 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:04.020019    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/773cfcaf-f677-4056-bbe8-1a475443aa1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dj7cl\" (UID: \"773cfcaf-f677-4056-bbe8-1a475443aa1b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl"
Sep 08 13:18:04.020218 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:04.020179    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 13:18:04.020264 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:04.020245    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/773cfcaf-f677-4056-bbe8-1a475443aa1b-cluster-monitoring-operator-tls podName:773cfcaf-f677-4056-bbe8-1a475443aa1b nodeName:}" failed. No retries permitted until 2026-09-08 13:18:08.020229971 +0000 UTC m=+119.480874759 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/773cfcaf-f677-4056-bbe8-1a475443aa1b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dj7cl" (UID: "773cfcaf-f677-4056-bbe8-1a475443aa1b") : secret "cluster-monitoring-operator-tls" not found
Sep 08 13:18:05.550952 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:05.550919    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hhmrh"]
Sep 08 13:18:05.553393 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:05.553369    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:05.564133 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:05.564108    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 13:18:05.564305 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:05.564166    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 13:18:05.564305 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:05.564224    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 13:18:05.564305 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:05.564165    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x5h6q\""
Sep 08 13:18:05.564497 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:05.564111    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 13:18:05.568307 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:05.568269    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hhmrh"]
Sep 08 13:18:05.633622 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:05.633534    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c20793a7-cc59-453b-9942-d527170ce508-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:05.633622 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:05.633615    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c20793a7-cc59-453b-9942-d527170ce508-data-volume\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:05.633842 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:05.633652    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c20793a7-cc59-453b-9942-d527170ce508-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:05.633842 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:05.633701    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5lps\" (UniqueName: \"kubernetes.io/projected/c20793a7-cc59-453b-9942-d527170ce508-kube-api-access-l5lps\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:05.633842 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:05.633768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c20793a7-cc59-453b-9942-d527170ce508-crio-socket\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:05.734897 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:05.734858    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c20793a7-cc59-453b-9942-d527170ce508-data-volume\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:05.734897 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:05.734897    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c20793a7-cc59-453b-9942-d527170ce508-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:05.735138 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:05.734935    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5lps\" (UniqueName: \"kubernetes.io/projected/c20793a7-cc59-453b-9942-d527170ce508-kube-api-access-l5lps\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:05.735138 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:05.734997    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c20793a7-cc59-453b-9942-d527170ce508-crio-socket\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:05.735138 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:05.735037    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:05.735138 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:05.735097    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:18:05.735138 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:05.735119    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c20793a7-cc59-453b-9942-d527170ce508-insights-runtime-extractor-tls podName:c20793a7-cc59-453b-9942-d527170ce508 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:06.235098005 +0000 UTC m=+117.695742800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c20793a7-cc59-453b-9942-d527170ce508-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hhmrh" (UID: "c20793a7-cc59-453b-9942-d527170ce508") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:05.735138 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:05.735043    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c20793a7-cc59-453b-9942-d527170ce508-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:05.735138 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:05.735134    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c20793a7-cc59-453b-9942-d527170ce508-crio-socket\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:05.735512 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:05.735153    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c20793a7-cc59-453b-9942-d527170ce508-kube-rbac-proxy-cm podName:c20793a7-cc59-453b-9942-d527170ce508 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:06.235136083 +0000 UTC m=+117.695780891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c20793a7-cc59-453b-9942-d527170ce508-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hhmrh" (UID: "c20793a7-cc59-453b-9942-d527170ce508") : configmap "kube-rbac-proxy" not found
Sep 08 13:18:05.735512 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:05.735222    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c20793a7-cc59-453b-9942-d527170ce508-data-volume\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:05.745878 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:05.745857    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5lps\" (UniqueName: \"kubernetes.io/projected/c20793a7-cc59-453b-9942-d527170ce508-kube-api-access-l5lps\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:06.240022 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:06.239985    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c20793a7-cc59-453b-9942-d527170ce508-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:06.240232 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:06.240040    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c20793a7-cc59-453b-9942-d527170ce508-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:06.240232 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:06.240122    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:18:06.240232 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:06.240204    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c20793a7-cc59-453b-9942-d527170ce508-kube-rbac-proxy-cm podName:c20793a7-cc59-453b-9942-d527170ce508 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:07.240189693 +0000 UTC m=+118.700834480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c20793a7-cc59-453b-9942-d527170ce508-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hhmrh" (UID: "c20793a7-cc59-453b-9942-d527170ce508") : configmap "kube-rbac-proxy" not found
Sep 08 13:18:06.240232 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:06.240129    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:06.240471 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:06.240273    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c20793a7-cc59-453b-9942-d527170ce508-insights-runtime-extractor-tls podName:c20793a7-cc59-453b-9942-d527170ce508 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:07.240257899 +0000 UTC m=+118.700902688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c20793a7-cc59-453b-9942-d527170ce508-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hhmrh" (UID: "c20793a7-cc59-453b-9942-d527170ce508") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:06.374487 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:06.374450    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-cvwkr" event={"ID":"549823a1-8d62-40a7-8de0-62cff329c97d","Type":"ContainerStarted","Data":"77db2d638c3ea04aaf42421bea4a08f7afd6cb05b834e3b35333fa594e31554c"}
Sep 08 13:18:06.376191 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:06.376161    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9q5lg" event={"ID":"509d01ad-7242-4f0f-9897-f0763e1b7298","Type":"ContainerStarted","Data":"888aa94e18f3149e948c313c720e6adeb44732e0b32baec72135bcba23e730ba"}
Sep 08 13:18:06.377803 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:06.377784    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/0.log"
Sep 08 13:18:06.377901 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:06.377819    2561 generic.go:356] "Generic (PLEG): container finished" podID="5e21ee11-3e84-4622-8cc7-4984c76637aa" containerID="865c695166fbcaee2bbc31b5a37093203a69597846d79a09d9711101861a39c4" exitCode=255
Sep 08 13:18:06.377901 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:06.377885    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-6g876" event={"ID":"5e21ee11-3e84-4622-8cc7-4984c76637aa","Type":"ContainerDied","Data":"865c695166fbcaee2bbc31b5a37093203a69597846d79a09d9711101861a39c4"}
Sep 08 13:18:06.378075 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:06.378061    2561 scope.go:117] "RemoveContainer" containerID="865c695166fbcaee2bbc31b5a37093203a69597846d79a09d9711101861a39c4"
Sep 08 13:18:06.379689 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:06.379658    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84cdcfc8d-v9vn7" event={"ID":"8e733f5a-3d38-4093-8f32-3fec4138aa41","Type":"ContainerStarted","Data":"041678c93910618c7c6ca5f4a58f32d21454ea853c9cbbb6b68a1278d08ae87e"}
Sep 08 13:18:06.391963 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:06.391916    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-cvwkr" podStartSLOduration=1.7582658119999999 podStartE2EDuration="6.391899692s" totalImagesPullingTime="4.63363388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:06.391033487 +0000 UTC m=+117.851678298" watchObservedRunningTime="2026-09-08 13:18:06.391899692 +0000 UTC m=+117.852544501"
Sep 08 13:18:06.425888 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:06.425813    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84cdcfc8d-v9vn7" podStartSLOduration=1.906333973 podStartE2EDuration="6.42579176s" totalImagesPullingTime="4.519457787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:06.424358222 +0000 UTC m=+117.885003032" watchObservedRunningTime="2026-09-08 13:18:06.42579176 +0000 UTC m=+117.886436571"
Sep 08 13:18:07.248974 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:07.248940    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c20793a7-cc59-453b-9942-d527170ce508-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:07.249411 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:07.249004    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c20793a7-cc59-453b-9942-d527170ce508-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:07.249411 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:07.249053    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:18:07.249411 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:07.249117    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c20793a7-cc59-453b-9942-d527170ce508-kube-rbac-proxy-cm podName:c20793a7-cc59-453b-9942-d527170ce508 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:09.24910214 +0000 UTC m=+120.709746933 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c20793a7-cc59-453b-9942-d527170ce508-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hhmrh" (UID: "c20793a7-cc59-453b-9942-d527170ce508") : configmap "kube-rbac-proxy" not found
Sep 08 13:18:07.249411 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:07.249153    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:07.249411 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:07.249209    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c20793a7-cc59-453b-9942-d527170ce508-insights-runtime-extractor-tls podName:c20793a7-cc59-453b-9942-d527170ce508 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:09.249195756 +0000 UTC m=+120.709840543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c20793a7-cc59-453b-9942-d527170ce508-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hhmrh" (UID: "c20793a7-cc59-453b-9942-d527170ce508") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:07.384683 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:07.384656    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 13:18:07.385135 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:07.385115    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/0.log"
Sep 08 13:18:07.385255 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:07.385158    2561 generic.go:356] "Generic (PLEG): container finished" podID="5e21ee11-3e84-4622-8cc7-4984c76637aa" containerID="ba313ec29d6fd475fd667bdc60eaedab4b220eb66ae52c9d971d9a21c2e67ae9" exitCode=255
Sep 08 13:18:07.385340 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:07.385249    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-6g876" event={"ID":"5e21ee11-3e84-4622-8cc7-4984c76637aa","Type":"ContainerDied","Data":"ba313ec29d6fd475fd667bdc60eaedab4b220eb66ae52c9d971d9a21c2e67ae9"}
Sep 08 13:18:07.385340 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:07.385327    2561 scope.go:117] "RemoveContainer" containerID="865c695166fbcaee2bbc31b5a37093203a69597846d79a09d9711101861a39c4"
Sep 08 13:18:07.385601 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:07.385582    2561 scope.go:117] "RemoveContainer" containerID="ba313ec29d6fd475fd667bdc60eaedab4b220eb66ae52c9d971d9a21c2e67ae9"
Sep 08 13:18:07.385800 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:07.385777    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-6g876_openshift-console-operator(5e21ee11-3e84-4622-8cc7-4984c76637aa)\"" pod="openshift-console-operator/console-operator-85b454874b-6g876" podUID="5e21ee11-3e84-4622-8cc7-4984c76637aa"
Sep 08 13:18:07.405306 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:07.403520    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9q5lg" podStartSLOduration=2.7606540649999998 podStartE2EDuration="7.403499942s" totalImagesPullingTime="4.642845877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:06.441732415 +0000 UTC m=+117.902377226" watchObservedRunningTime="2026-09-08 13:18:07.403499942 +0000 UTC m=+118.864144752"
Sep 08 13:18:07.492658 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:07.492627    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t8k5r_bb02bae4-4bf7-4018-b0a3-e9f65b5057fb/dns-node-resolver/0.log"
Sep 08 13:18:07.955911 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:07.955866    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-tls\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:07.956108 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:07.956007    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:18:07.956108 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:07.956025    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5457d499bd-hknkx: secret "image-registry-tls" not found
Sep 08 13:18:07.956108 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:07.956089    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-tls podName:ac87abf4-bf58-4f6b-897c-ed20252c6b33 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:15.956070171 +0000 UTC m=+127.416714973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-tls") pod "image-registry-5457d499bd-hknkx" (UID: "ac87abf4-bf58-4f6b-897c-ed20252c6b33") : secret "image-registry-tls" not found
Sep 08 13:18:08.057066 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:08.057013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/773cfcaf-f677-4056-bbe8-1a475443aa1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dj7cl\" (UID: \"773cfcaf-f677-4056-bbe8-1a475443aa1b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl"
Sep 08 13:18:08.057239 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:08.057162    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 13:18:08.057239 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:08.057227    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/773cfcaf-f677-4056-bbe8-1a475443aa1b-cluster-monitoring-operator-tls podName:773cfcaf-f677-4056-bbe8-1a475443aa1b nodeName:}" failed. No retries permitted until 2026-09-08 13:18:16.057210243 +0000 UTC m=+127.517855053 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/773cfcaf-f677-4056-bbe8-1a475443aa1b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dj7cl" (UID: "773cfcaf-f677-4056-bbe8-1a475443aa1b") : secret "cluster-monitoring-operator-tls" not found
Sep 08 13:18:08.390637 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:08.390555    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 13:18:08.390991 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:08.390883    2561 scope.go:117] "RemoveContainer" containerID="ba313ec29d6fd475fd667bdc60eaedab4b220eb66ae52c9d971d9a21c2e67ae9"
Sep 08 13:18:08.391058 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:08.391040    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-6g876_openshift-console-operator(5e21ee11-3e84-4622-8cc7-4984c76637aa)\"" pod="openshift-console-operator/console-operator-85b454874b-6g876" podUID="5e21ee11-3e84-4622-8cc7-4984c76637aa"
Sep 08 13:18:08.695011 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:08.694988    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f4gjd_fca118ca-6b4f-4c1b-9fa7-b384f31387a4/node-ca/0.log"
Sep 08 13:18:09.267923 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:09.267862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c20793a7-cc59-453b-9942-d527170ce508-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:09.268112 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:09.267988    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c20793a7-cc59-453b-9942-d527170ce508-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:09.268112 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:09.268039    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:09.268112 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:09.268076    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:18:09.268112 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:09.268110    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c20793a7-cc59-453b-9942-d527170ce508-insights-runtime-extractor-tls podName:c20793a7-cc59-453b-9942-d527170ce508 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:13.268093157 +0000 UTC m=+124.728737946 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c20793a7-cc59-453b-9942-d527170ce508-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hhmrh" (UID: "c20793a7-cc59-453b-9942-d527170ce508") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:09.268254 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:09.268125    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c20793a7-cc59-453b-9942-d527170ce508-kube-rbac-proxy-cm podName:c20793a7-cc59-453b-9942-d527170ce508 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:13.268118157 +0000 UTC m=+124.728762945 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c20793a7-cc59-453b-9942-d527170ce508-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hhmrh" (UID: "c20793a7-cc59-453b-9942-d527170ce508") : configmap "kube-rbac-proxy" not found
Sep 08 13:18:09.294502 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:09.294476    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q5qrk_a54b8b62-55df-4e1c-a738-ae5081203413/serve-healthcheck-canary/0.log"
Sep 08 13:18:10.188207 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.188171    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-wkcqw"]
Sep 08 13:18:10.191157 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.191140    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-wkcqw"
Sep 08 13:18:10.193539 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.193519    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8dcjn\""
Sep 08 13:18:10.193657 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.193580    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 08 13:18:10.194237 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.194212    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 08 13:18:10.194351 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.194241    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 08 13:18:10.194351 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.194249    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 08 13:18:10.199941 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.199918    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-wkcqw"]
Sep 08 13:18:10.275309 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.275250    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b3e8b73d-8db9-47b8-8127-519a61215387-signing-key\") pod \"service-ca-75d99c484-wkcqw\" (UID: \"b3e8b73d-8db9-47b8-8127-519a61215387\") " pod="openshift-service-ca/service-ca-75d99c484-wkcqw"
Sep 08 13:18:10.275498 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.275322    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b3e8b73d-8db9-47b8-8127-519a61215387-signing-cabundle\") pod \"service-ca-75d99c484-wkcqw\" (UID: \"b3e8b73d-8db9-47b8-8127-519a61215387\") " pod="openshift-service-ca/service-ca-75d99c484-wkcqw"
Sep 08 13:18:10.275498 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.275452    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk7lp\" (UniqueName: \"kubernetes.io/projected/b3e8b73d-8db9-47b8-8127-519a61215387-kube-api-access-hk7lp\") pod \"service-ca-75d99c484-wkcqw\" (UID: \"b3e8b73d-8db9-47b8-8127-519a61215387\") " pod="openshift-service-ca/service-ca-75d99c484-wkcqw"
Sep 08 13:18:10.376512 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.376475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b3e8b73d-8db9-47b8-8127-519a61215387-signing-cabundle\") pod \"service-ca-75d99c484-wkcqw\" (UID: \"b3e8b73d-8db9-47b8-8127-519a61215387\") " pod="openshift-service-ca/service-ca-75d99c484-wkcqw"
Sep 08 13:18:10.376702 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.376542    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk7lp\" (UniqueName: \"kubernetes.io/projected/b3e8b73d-8db9-47b8-8127-519a61215387-kube-api-access-hk7lp\") pod \"service-ca-75d99c484-wkcqw\" (UID: \"b3e8b73d-8db9-47b8-8127-519a61215387\") " pod="openshift-service-ca/service-ca-75d99c484-wkcqw"
Sep 08 13:18:10.376702 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.376606    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b3e8b73d-8db9-47b8-8127-519a61215387-signing-key\") pod \"service-ca-75d99c484-wkcqw\" (UID: \"b3e8b73d-8db9-47b8-8127-519a61215387\") " pod="openshift-service-ca/service-ca-75d99c484-wkcqw"
Sep 08 13:18:10.377101 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.377077    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b3e8b73d-8db9-47b8-8127-519a61215387-signing-cabundle\") pod \"service-ca-75d99c484-wkcqw\" (UID: \"b3e8b73d-8db9-47b8-8127-519a61215387\") " pod="openshift-service-ca/service-ca-75d99c484-wkcqw"
Sep 08 13:18:10.378932 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.378910    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b3e8b73d-8db9-47b8-8127-519a61215387-signing-key\") pod \"service-ca-75d99c484-wkcqw\" (UID: \"b3e8b73d-8db9-47b8-8127-519a61215387\") " pod="openshift-service-ca/service-ca-75d99c484-wkcqw"
Sep 08 13:18:10.385311 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.385278    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk7lp\" (UniqueName: \"kubernetes.io/projected/b3e8b73d-8db9-47b8-8127-519a61215387-kube-api-access-hk7lp\") pod \"service-ca-75d99c484-wkcqw\" (UID: \"b3e8b73d-8db9-47b8-8127-519a61215387\") " pod="openshift-service-ca/service-ca-75d99c484-wkcqw"
Sep 08 13:18:10.471736 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.471636    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-6g876"
Sep 08 13:18:10.471736 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.471712    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-6g876"
Sep 08 13:18:10.472081 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.472068    2561 scope.go:117] "RemoveContainer" containerID="ba313ec29d6fd475fd667bdc60eaedab4b220eb66ae52c9d971d9a21c2e67ae9"
Sep 08 13:18:10.472245 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:10.472228    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-6g876_openshift-console-operator(5e21ee11-3e84-4622-8cc7-4984c76637aa)\"" pod="openshift-console-operator/console-operator-85b454874b-6g876" podUID="5e21ee11-3e84-4622-8cc7-4984c76637aa"
Sep 08 13:18:10.499907 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.499870    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-wkcqw"
Sep 08 13:18:10.776346 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:10.776318    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-wkcqw"]
Sep 08 13:18:10.776805 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:18:10.776779    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3e8b73d_8db9_47b8_8127_519a61215387.slice/crio-3d3516004cced09dc526d651d7210b3dc79f8dc59105cf34067d437ece8905d8 WatchSource:0}: Error finding container 3d3516004cced09dc526d651d7210b3dc79f8dc59105cf34067d437ece8905d8: Status 404 returned error can't find the container with id 3d3516004cced09dc526d651d7210b3dc79f8dc59105cf34067d437ece8905d8
Sep 08 13:18:11.403779 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:11.403738    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-wkcqw" event={"ID":"b3e8b73d-8db9-47b8-8127-519a61215387","Type":"ContainerStarted","Data":"439dcb9e5f7f6aa72c7d08fd2cacf7ac11867dcf03a35bf5100a46b035daff73"}
Sep 08 13:18:11.404211 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:11.403789    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-wkcqw" event={"ID":"b3e8b73d-8db9-47b8-8127-519a61215387","Type":"ContainerStarted","Data":"3d3516004cced09dc526d651d7210b3dc79f8dc59105cf34067d437ece8905d8"}
Sep 08 13:18:13.301195 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:13.301150    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c20793a7-cc59-453b-9942-d527170ce508-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:13.301638 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:13.301212    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c20793a7-cc59-453b-9942-d527170ce508-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:13.301638 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:13.301305    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:18:13.301638 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:13.301333    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:13.301638 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:13.301394    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c20793a7-cc59-453b-9942-d527170ce508-kube-rbac-proxy-cm podName:c20793a7-cc59-453b-9942-d527170ce508 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:21.301367328 +0000 UTC m=+132.762012118 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c20793a7-cc59-453b-9942-d527170ce508-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hhmrh" (UID: "c20793a7-cc59-453b-9942-d527170ce508") : configmap "kube-rbac-proxy" not found
Sep 08 13:18:13.301638 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:13.301418    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c20793a7-cc59-453b-9942-d527170ce508-insights-runtime-extractor-tls podName:c20793a7-cc59-453b-9942-d527170ce508 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:21.301407968 +0000 UTC m=+132.762052777 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c20793a7-cc59-453b-9942-d527170ce508-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hhmrh" (UID: "c20793a7-cc59-453b-9942-d527170ce508") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:16.023615 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:16.023566    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-tls\") pod \"image-registry-5457d499bd-hknkx\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") " pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:16.024029 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:16.023729    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:18:16.024029 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:16.023752    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5457d499bd-hknkx: secret "image-registry-tls" not found
Sep 08 13:18:16.024029 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:16.023809    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-tls podName:ac87abf4-bf58-4f6b-897c-ed20252c6b33 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:32.023792553 +0000 UTC m=+143.484437357 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-tls") pod "image-registry-5457d499bd-hknkx" (UID: "ac87abf4-bf58-4f6b-897c-ed20252c6b33") : secret "image-registry-tls" not found
Sep 08 13:18:16.124722 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:16.124676    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/773cfcaf-f677-4056-bbe8-1a475443aa1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dj7cl\" (UID: \"773cfcaf-f677-4056-bbe8-1a475443aa1b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl"
Sep 08 13:18:16.124895 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:16.124827    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 13:18:16.124895 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:16.124892    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/773cfcaf-f677-4056-bbe8-1a475443aa1b-cluster-monitoring-operator-tls podName:773cfcaf-f677-4056-bbe8-1a475443aa1b nodeName:}" failed. No retries permitted until 2026-09-08 13:18:32.124875222 +0000 UTC m=+143.585520025 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/773cfcaf-f677-4056-bbe8-1a475443aa1b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dj7cl" (UID: "773cfcaf-f677-4056-bbe8-1a475443aa1b") : secret "cluster-monitoring-operator-tls" not found
Sep 08 13:18:18.745665 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:18.745626    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs\") pod \"network-metrics-daemon-gsgmd\" (UID: \"246d684e-5ff4-49bb-b39c-660e46385842\") " pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:18:18.746096 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:18.745813    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 13:18:18.746096 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:18.745894    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs podName:246d684e-5ff4-49bb-b39c-660e46385842 nodeName:}" failed. No retries permitted until 2026-09-08 13:20:20.745874255 +0000 UTC m=+252.206519059 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs") pod "network-metrics-daemon-gsgmd" (UID: "246d684e-5ff4-49bb-b39c-660e46385842") : secret "metrics-daemon-secret" not found
Sep 08 13:18:21.369339 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:21.369279    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c20793a7-cc59-453b-9942-d527170ce508-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:21.369815 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:21.369358    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c20793a7-cc59-453b-9942-d527170ce508-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:21.369815 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:21.369460    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:21.369815 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:21.369530    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c20793a7-cc59-453b-9942-d527170ce508-insights-runtime-extractor-tls podName:c20793a7-cc59-453b-9942-d527170ce508 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:37.369512835 +0000 UTC m=+148.830157623 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c20793a7-cc59-453b-9942-d527170ce508-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hhmrh" (UID: "c20793a7-cc59-453b-9942-d527170ce508") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:21.370015 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:21.369991    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c20793a7-cc59-453b-9942-d527170ce508-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:23.000486 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:23.000454    2561 scope.go:117] "RemoveContainer" containerID="ba313ec29d6fd475fd667bdc60eaedab4b220eb66ae52c9d971d9a21c2e67ae9"
Sep 08 13:18:23.441499 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:23.441470    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 13:18:23.441670 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:23.441528    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-6g876" event={"ID":"5e21ee11-3e84-4622-8cc7-4984c76637aa","Type":"ContainerStarted","Data":"0a63f39f6dfda8f1b1d717977bdfdee606251fca7013e06b33e3e4b29fd43ce9"}
Sep 08 13:18:23.441814 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:23.441797    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-6g876"
Sep 08 13:18:23.459429 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:23.459374    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-6g876" podStartSLOduration=18.848220936 podStartE2EDuration="23.459359378s" totalImagesPullingTime="4.611138442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:23.458989318 +0000 UTC m=+134.919634122" watchObservedRunningTime="2026-09-08 13:18:23.459359378 +0000 UTC m=+134.920004211"
Sep 08 13:18:23.459601 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:23.459514    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-wkcqw" podStartSLOduration=13.459508837 podStartE2EDuration="13.459508837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:11.423488731 +0000 UTC m=+122.884133540" watchObservedRunningTime="2026-09-08 13:18:23.459508837 +0000 UTC m=+134.920153647"
Sep 08 13:18:23.905122 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:23.905043    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-6g876"
Sep 08 13:18:29.205245 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.205209    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5457d499bd-hknkx"]
Sep 08 13:18:29.205629 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:29.205461    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5457d499bd-hknkx" podUID="ac87abf4-bf58-4f6b-897c-ed20252c6b33"
Sep 08 13:18:29.238953 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.238921    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b96b98575-gslm8"]
Sep 08 13:18:29.241348 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.241325    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.259961 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.259930    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b96b98575-gslm8"]
Sep 08 13:18:29.331020 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.330986    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c21fa545-edea-4e71-b403-f68625c83518-image-registry-private-configuration\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.331020 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.331022    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c21fa545-edea-4e71-b403-f68625c83518-installation-pull-secrets\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.331251 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.331044    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtzr2\" (UniqueName: \"kubernetes.io/projected/c21fa545-edea-4e71-b403-f68625c83518-kube-api-access-jtzr2\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.331251 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.331115    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c21fa545-edea-4e71-b403-f68625c83518-registry-certificates\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.331251 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.331237    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c21fa545-edea-4e71-b403-f68625c83518-ca-trust-extracted\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.331441 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.331311    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c21fa545-edea-4e71-b403-f68625c83518-trusted-ca\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.331441 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.331368    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c21fa545-edea-4e71-b403-f68625c83518-registry-tls\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.331441 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.331396    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c21fa545-edea-4e71-b403-f68625c83518-bound-sa-token\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.431787 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.431749    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c21fa545-edea-4e71-b403-f68625c83518-image-registry-private-configuration\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.431787 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.431789    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c21fa545-edea-4e71-b403-f68625c83518-installation-pull-secrets\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.432072 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.431810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtzr2\" (UniqueName: \"kubernetes.io/projected/c21fa545-edea-4e71-b403-f68625c83518-kube-api-access-jtzr2\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.432072 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.431843    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c21fa545-edea-4e71-b403-f68625c83518-registry-certificates\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.432072 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.431897    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c21fa545-edea-4e71-b403-f68625c83518-ca-trust-extracted\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.432072 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.431935    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c21fa545-edea-4e71-b403-f68625c83518-trusted-ca\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.432072 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.431986    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c21fa545-edea-4e71-b403-f68625c83518-registry-tls\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.432072 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.432032    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c21fa545-edea-4e71-b403-f68625c83518-bound-sa-token\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.432422 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.432387    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c21fa545-edea-4e71-b403-f68625c83518-ca-trust-extracted\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.432839 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.432813    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c21fa545-edea-4e71-b403-f68625c83518-registry-certificates\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.433144 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.433121    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c21fa545-edea-4e71-b403-f68625c83518-trusted-ca\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.434671 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.434643    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c21fa545-edea-4e71-b403-f68625c83518-registry-tls\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.434755 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.434707    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c21fa545-edea-4e71-b403-f68625c83518-installation-pull-secrets\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.434801 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.434759    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c21fa545-edea-4e71-b403-f68625c83518-image-registry-private-configuration\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.441053 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.441020    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtzr2\" (UniqueName: \"kubernetes.io/projected/c21fa545-edea-4e71-b403-f68625c83518-kube-api-access-jtzr2\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.443149 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.443124    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c21fa545-edea-4e71-b403-f68625c83518-bound-sa-token\") pod \"image-registry-b96b98575-gslm8\" (UID: \"c21fa545-edea-4e71-b403-f68625c83518\") " pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.459379 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.459325    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:29.465718 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.465601    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:29.552030 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.551994    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wqhr2\""
Sep 08 13:18:29.560617 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.560586    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:29.636710 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.636676    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac87abf4-bf58-4f6b-897c-ed20252c6b33-installation-pull-secrets\") pod \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") "
Sep 08 13:18:29.636939 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.636919    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac87abf4-bf58-4f6b-897c-ed20252c6b33-image-registry-private-configuration\") pod \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") "
Sep 08 13:18:29.637070 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.637055    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5pxd\" (UniqueName: \"kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-kube-api-access-g5pxd\") pod \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") "
Sep 08 13:18:29.637203 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.637188    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac87abf4-bf58-4f6b-897c-ed20252c6b33-ca-trust-extracted\") pod \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") "
Sep 08 13:18:29.637342 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.637327    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-bound-sa-token\") pod \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") "
Sep 08 13:18:29.637440 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.637420    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-certificates\") pod \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") "
Sep 08 13:18:29.637500 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.637464    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac87abf4-bf58-4f6b-897c-ed20252c6b33-trusted-ca\") pod \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\" (UID: \"ac87abf4-bf58-4f6b-897c-ed20252c6b33\") "
Sep 08 13:18:29.638243 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.638214    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac87abf4-bf58-4f6b-897c-ed20252c6b33-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ac87abf4-bf58-4f6b-897c-ed20252c6b33" (UID: "ac87abf4-bf58-4f6b-897c-ed20252c6b33"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:18:29.639584 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.639542    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ac87abf4-bf58-4f6b-897c-ed20252c6b33" (UID: "ac87abf4-bf58-4f6b-897c-ed20252c6b33"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:18:29.641985 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.641937    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-kube-api-access-g5pxd" (OuterVolumeSpecName: "kube-api-access-g5pxd") pod "ac87abf4-bf58-4f6b-897c-ed20252c6b33" (UID: "ac87abf4-bf58-4f6b-897c-ed20252c6b33"). InnerVolumeSpecName "kube-api-access-g5pxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:18:29.642226 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.642186    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ac87abf4-bf58-4f6b-897c-ed20252c6b33" (UID: "ac87abf4-bf58-4f6b-897c-ed20252c6b33"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:18:29.643192 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.643161    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac87abf4-bf58-4f6b-897c-ed20252c6b33-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ac87abf4-bf58-4f6b-897c-ed20252c6b33" (UID: "ac87abf4-bf58-4f6b-897c-ed20252c6b33"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:18:29.643503 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.643479    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac87abf4-bf58-4f6b-897c-ed20252c6b33-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ac87abf4-bf58-4f6b-897c-ed20252c6b33" (UID: "ac87abf4-bf58-4f6b-897c-ed20252c6b33"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:18:29.644637 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.644616    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac87abf4-bf58-4f6b-897c-ed20252c6b33-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ac87abf4-bf58-4f6b-897c-ed20252c6b33" (UID: "ac87abf4-bf58-4f6b-897c-ed20252c6b33"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:18:29.738435 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.738349    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-certificates\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:18:29.738435 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.738378    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac87abf4-bf58-4f6b-897c-ed20252c6b33-trusted-ca\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:18:29.738435 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.738387    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac87abf4-bf58-4f6b-897c-ed20252c6b33-installation-pull-secrets\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:18:29.738435 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.738397    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac87abf4-bf58-4f6b-897c-ed20252c6b33-image-registry-private-configuration\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:18:29.738435 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.738407    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5pxd\" (UniqueName: \"kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-kube-api-access-g5pxd\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:18:29.738435 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.738416    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac87abf4-bf58-4f6b-897c-ed20252c6b33-ca-trust-extracted\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:18:29.738435 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.738425    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-bound-sa-token\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:18:29.843909 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:29.843878    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b96b98575-gslm8"]
Sep 08 13:18:29.847239 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:18:29.847214    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21fa545_edea_4e71_b403_f68625c83518.slice/crio-7add861049e52e74171069af04ecad60a67530c802758c602606b0c410008b48 WatchSource:0}: Error finding container 7add861049e52e74171069af04ecad60a67530c802758c602606b0c410008b48: Status 404 returned error can't find the container with id 7add861049e52e74171069af04ecad60a67530c802758c602606b0c410008b48
Sep 08 13:18:30.464776 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:30.464740    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b96b98575-gslm8" event={"ID":"c21fa545-edea-4e71-b403-f68625c83518","Type":"ContainerStarted","Data":"4fcf2f4a53a9d5c1552365944be46808be85f42b879ca895c35ebc21e23b6309"}
Sep 08 13:18:30.465141 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:30.464790    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b96b98575-gslm8" event={"ID":"c21fa545-edea-4e71-b403-f68625c83518","Type":"ContainerStarted","Data":"7add861049e52e74171069af04ecad60a67530c802758c602606b0c410008b48"}
Sep 08 13:18:30.465141 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:30.464820    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:30.465141 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:30.464916    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5457d499bd-hknkx"
Sep 08 13:18:30.492377 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:30.492324    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b96b98575-gslm8" podStartSLOduration=1.492307134 podStartE2EDuration="1.492307134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:30.490394314 +0000 UTC m=+141.951039126" watchObservedRunningTime="2026-09-08 13:18:30.492307134 +0000 UTC m=+141.952951969"
Sep 08 13:18:30.519507 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:30.519476    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5457d499bd-hknkx"]
Sep 08 13:18:30.525868 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:30.525842    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5457d499bd-hknkx"]
Sep 08 13:18:30.647150 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:30.647099    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac87abf4-bf58-4f6b-897c-ed20252c6b33-registry-tls\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:18:31.004899 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:31.004868    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac87abf4-bf58-4f6b-897c-ed20252c6b33" path="/var/lib/kubelet/pods/ac87abf4-bf58-4f6b-897c-ed20252c6b33/volumes"
Sep 08 13:18:32.159001 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:32.158948    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/773cfcaf-f677-4056-bbe8-1a475443aa1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dj7cl\" (UID: \"773cfcaf-f677-4056-bbe8-1a475443aa1b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl"
Sep 08 13:18:32.161358 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:32.161338    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/773cfcaf-f677-4056-bbe8-1a475443aa1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dj7cl\" (UID: \"773cfcaf-f677-4056-bbe8-1a475443aa1b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl"
Sep 08 13:18:32.450785 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:32.450752    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-m5dl8\""
Sep 08 13:18:32.459557 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:32.459532    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl"
Sep 08 13:18:32.731578 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:32.731505    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl"]
Sep 08 13:18:32.734546 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:18:32.734520    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773cfcaf_f677_4056_bbe8_1a475443aa1b.slice/crio-da72a8a3ac63c9ea0c737a9febf32c1b9ca26d13e0e59c4e19354ff02ccf4ba8 WatchSource:0}: Error finding container da72a8a3ac63c9ea0c737a9febf32c1b9ca26d13e0e59c4e19354ff02ccf4ba8: Status 404 returned error can't find the container with id da72a8a3ac63c9ea0c737a9febf32c1b9ca26d13e0e59c4e19354ff02ccf4ba8
Sep 08 13:18:33.477634 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:33.477599    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl" event={"ID":"773cfcaf-f677-4056-bbe8-1a475443aa1b","Type":"ContainerStarted","Data":"da72a8a3ac63c9ea0c737a9febf32c1b9ca26d13e0e59c4e19354ff02ccf4ba8"}
Sep 08 13:18:37.396448 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:37.396401    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c20793a7-cc59-453b-9942-d527170ce508-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:37.399021 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:37.398992    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c20793a7-cc59-453b-9942-d527170ce508-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hhmrh\" (UID: \"c20793a7-cc59-453b-9942-d527170ce508\") " pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:37.411400 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:37.411372    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x5h6q\""
Sep 08 13:18:37.419448 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:37.419420    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hhmrh"
Sep 08 13:18:37.718705 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:37.718668    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hhmrh"]
Sep 08 13:18:37.721984 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:18:37.721954    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc20793a7_cc59_453b_9942_d527170ce508.slice/crio-893f4d84b1325071257c90e672bbf5f05415aec7fcaa1dc44a08010d6510ce21 WatchSource:0}: Error finding container 893f4d84b1325071257c90e672bbf5f05415aec7fcaa1dc44a08010d6510ce21: Status 404 returned error can't find the container with id 893f4d84b1325071257c90e672bbf5f05415aec7fcaa1dc44a08010d6510ce21
Sep 08 13:18:38.495663 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:38.495620    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hhmrh" event={"ID":"c20793a7-cc59-453b-9942-d527170ce508","Type":"ContainerStarted","Data":"0dfff638f1ea308ca1ce997ec50bb9c74a73b0f52a37ee66ee130994ee95614f"}
Sep 08 13:18:38.496136 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:38.495693    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hhmrh" event={"ID":"c20793a7-cc59-453b-9942-d527170ce508","Type":"ContainerStarted","Data":"893f4d84b1325071257c90e672bbf5f05415aec7fcaa1dc44a08010d6510ce21"}
Sep 08 13:18:39.500769 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:39.500712    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl" event={"ID":"773cfcaf-f677-4056-bbe8-1a475443aa1b","Type":"ContainerStarted","Data":"9220b768b337c54d950b419204450d92e7e9c1457b12c97b6ea258016b5ced78"}
Sep 08 13:18:39.519573 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:39.519511    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dj7cl" podStartSLOduration=32.927739169 podStartE2EDuration="39.519490534s" totalImagesPullingTime="6.591751365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:39.519034029 +0000 UTC m=+150.979678838" watchObservedRunningTime="2026-09-08 13:18:39.519490534 +0000 UTC m=+150.980135340"
Sep 08 13:18:42.511526 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:42.511485    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hhmrh" event={"ID":"c20793a7-cc59-453b-9942-d527170ce508","Type":"ContainerStarted","Data":"7c86b649dea3be08ed95dd24f69348bbd73b7079976304cdb8c801ce423962b5"}
Sep 08 13:18:44.389786 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.389750    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cc4fbd768-kwxdc"]
Sep 08 13:18:44.391906 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.391886    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.395132 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.395103    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 08 13:18:44.395132 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.395104    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 08 13:18:44.395505 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.395487    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 08 13:18:44.395588 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.395493    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 08 13:18:44.395653 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.395587    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 08 13:18:44.396002 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.395987    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kzhlx\""
Sep 08 13:18:44.396084 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.395988    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 08 13:18:44.396084 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.396076    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 08 13:18:44.400881 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.400861    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 08 13:18:44.405323 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.405301    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cc4fbd768-kwxdc"]
Sep 08 13:18:44.448148 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.448103    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-oauth-serving-cert\") pod \"console-cc4fbd768-kwxdc\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") " pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.448148 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.448147    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb6mq\" (UniqueName: \"kubernetes.io/projected/db680cae-3410-41eb-a095-432aee141598-kube-api-access-lb6mq\") pod \"console-cc4fbd768-kwxdc\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") " pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.448428 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.448192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-console-config\") pod \"console-cc4fbd768-kwxdc\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") " pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.448428 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.448214    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-service-ca\") pod \"console-cc4fbd768-kwxdc\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") " pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.448428 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.448338    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db680cae-3410-41eb-a095-432aee141598-console-serving-cert\") pod \"console-cc4fbd768-kwxdc\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") " pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.448428 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.448375    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-trusted-ca-bundle\") pod \"console-cc4fbd768-kwxdc\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") " pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.448428 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.448409    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db680cae-3410-41eb-a095-432aee141598-console-oauth-config\") pod \"console-cc4fbd768-kwxdc\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") " pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.520973 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.520937    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hhmrh" event={"ID":"c20793a7-cc59-453b-9942-d527170ce508","Type":"ContainerStarted","Data":"6fdd5b2c6aed4a65b94026358c553070e9da1e3a341654642384c3428b90d343"}
Sep 08 13:18:44.541343 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.541275    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hhmrh" podStartSLOduration=33.878259807 podStartE2EDuration="39.54125898s" totalImagesPullingTime="5.662999173s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:18:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:44.53986321 +0000 UTC m=+156.000508037" watchObservedRunningTime="2026-09-08 13:18:44.54125898 +0000 UTC m=+156.001903834"
Sep 08 13:18:44.549121 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.549095    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db680cae-3410-41eb-a095-432aee141598-console-oauth-config\") pod \"console-cc4fbd768-kwxdc\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") " pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.549207 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.549137    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-oauth-serving-cert\") pod \"console-cc4fbd768-kwxdc\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") " pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.549207 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.549153    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb6mq\" (UniqueName: \"kubernetes.io/projected/db680cae-3410-41eb-a095-432aee141598-kube-api-access-lb6mq\") pod \"console-cc4fbd768-kwxdc\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") " pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.549342 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.549318    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-console-config\") pod \"console-cc4fbd768-kwxdc\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") " pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.549399 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.549379    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-service-ca\") pod \"console-cc4fbd768-kwxdc\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") " pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.549539 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.549452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db680cae-3410-41eb-a095-432aee141598-console-serving-cert\") pod \"console-cc4fbd768-kwxdc\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") " pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.549539 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.549479    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-trusted-ca-bundle\") pod \"console-cc4fbd768-kwxdc\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") " pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.550008 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.549981    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-console-config\") pod \"console-cc4fbd768-kwxdc\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") " pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.550128 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.550066    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-service-ca\") pod \"console-cc4fbd768-kwxdc\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") " pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.550503 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.550482    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-trusted-ca-bundle\") pod \"console-cc4fbd768-kwxdc\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") " pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.550597 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.550512    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-oauth-serving-cert\") pod \"console-cc4fbd768-kwxdc\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") " pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.551683 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.551655    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db680cae-3410-41eb-a095-432aee141598-console-oauth-config\") pod \"console-cc4fbd768-kwxdc\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") " pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.551833 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.551815    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db680cae-3410-41eb-a095-432aee141598-console-serving-cert\") pod \"console-cc4fbd768-kwxdc\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") " pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.557693 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.557673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb6mq\" (UniqueName: \"kubernetes.io/projected/db680cae-3410-41eb-a095-432aee141598-kube-api-access-lb6mq\") pod \"console-cc4fbd768-kwxdc\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") " pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.701664 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.701632    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:44.975877 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:44.975802    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cc4fbd768-kwxdc"]
Sep 08 13:18:44.978758 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:18:44.978727    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb680cae_3410_41eb_a095_432aee141598.slice/crio-be3bb7d645b75d2e60cd0388e816d2d95a04925a8881de87ac5133e719675a7c WatchSource:0}: Error finding container be3bb7d645b75d2e60cd0388e816d2d95a04925a8881de87ac5133e719675a7c: Status 404 returned error can't find the container with id be3bb7d645b75d2e60cd0388e816d2d95a04925a8881de87ac5133e719675a7c
Sep 08 13:18:45.375943 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:45.375846    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-242sh" podUID="dcfe8764-419e-438f-91fb-c2f3892df1b2"
Sep 08 13:18:45.525481 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:45.525452    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-242sh"
Sep 08 13:18:45.525919 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:45.525451    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc4fbd768-kwxdc" event={"ID":"db680cae-3410-41eb-a095-432aee141598","Type":"ContainerStarted","Data":"be3bb7d645b75d2e60cd0388e816d2d95a04925a8881de87ac5133e719675a7c"}
Sep 08 13:18:46.027014 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:46.026966    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-gsgmd" podUID="246d684e-5ff4-49bb-b39c-660e46385842"
Sep 08 13:18:50.295472 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:50.295429    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls\") pod \"dns-default-242sh\" (UID: \"dcfe8764-419e-438f-91fb-c2f3892df1b2\") " pod="openshift-dns/dns-default-242sh"
Sep 08 13:18:50.297868 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:50.297842    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dcfe8764-419e-438f-91fb-c2f3892df1b2-metrics-tls\") pod \"dns-default-242sh\" (UID: \"dcfe8764-419e-438f-91fb-c2f3892df1b2\") " pod="openshift-dns/dns-default-242sh"
Sep 08 13:18:50.329378 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:50.329340    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8shlb\""
Sep 08 13:18:50.337399 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:50.337361    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-242sh"
Sep 08 13:18:51.183380 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:51.183278    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-242sh"]
Sep 08 13:18:51.186938 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:18:51.186906    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcfe8764_419e_438f_91fb_c2f3892df1b2.slice/crio-9dd4ea3e3e17c062f80ddc0db600cf420645d58375ddae5706df77a8d0311aac WatchSource:0}: Error finding container 9dd4ea3e3e17c062f80ddc0db600cf420645d58375ddae5706df77a8d0311aac: Status 404 returned error can't find the container with id 9dd4ea3e3e17c062f80ddc0db600cf420645d58375ddae5706df77a8d0311aac
Sep 08 13:18:51.473497 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:51.473467    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b96b98575-gslm8"
Sep 08 13:18:51.549378 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:51.549337    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc4fbd768-kwxdc" event={"ID":"db680cae-3410-41eb-a095-432aee141598","Type":"ContainerStarted","Data":"ab319bb240d50bd4b19fa907f33cb22e7d6dbc999a1566ab5154436f0baef0d8"}
Sep 08 13:18:51.550757 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:51.550724    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-242sh" event={"ID":"dcfe8764-419e-438f-91fb-c2f3892df1b2","Type":"ContainerStarted","Data":"9dd4ea3e3e17c062f80ddc0db600cf420645d58375ddae5706df77a8d0311aac"}
Sep 08 13:18:51.571727 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:51.571675    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cc4fbd768-kwxdc" podStartSLOduration=1.606897043 podStartE2EDuration="7.571657505s" totalImagesPullingTime="5.964760462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:51.570754305 +0000 UTC m=+163.031399116" watchObservedRunningTime="2026-09-08 13:18:51.571657505 +0000 UTC m=+163.032302315"
Sep 08 13:18:53.564336 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:53.564299    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-242sh" event={"ID":"dcfe8764-419e-438f-91fb-c2f3892df1b2","Type":"ContainerStarted","Data":"b36091b3f8d8842f204d6234f7f6cb9966a9548f7e5f4eb6a4db2963d4a21ffb"}
Sep 08 13:18:53.564726 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:53.564353    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-242sh" event={"ID":"dcfe8764-419e-438f-91fb-c2f3892df1b2","Type":"ContainerStarted","Data":"e6d79b3650616330c78b72a2cf18f0eb390f9f50e3dbfed79cee1ab3e709bef3"}
Sep 08 13:18:53.564726 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:53.564447    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-242sh"
Sep 08 13:18:53.586798 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:53.586749    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-242sh" podStartSLOduration=129.987504608 podStartE2EDuration="2m11.586732674s" totalImagesPullingTime="1.599228066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:53.586253896 +0000 UTC m=+165.046898705" watchObservedRunningTime="2026-09-08 13:18:53.586732674 +0000 UTC m=+165.047377483"
Sep 08 13:18:54.702399 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:54.702357    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:54.702860 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:54.702434    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:54.707268 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:54.707244    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:55.575663 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:55.575635    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:18:56.344940 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.344901    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq"]
Sep 08 13:18:56.348761 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.348737    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq"
Sep 08 13:18:56.356307 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.352270    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 13:18:56.356307 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.352333    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-f2f5f\""
Sep 08 13:18:56.356307 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.352534    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 08 13:18:56.356307 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.352952    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 08 13:18:56.362271 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.362234    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq"]
Sep 08 13:18:56.374948 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.374915    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q7vd2"]
Sep 08 13:18:56.378695 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.378669    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.380612 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.380588    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"]
Sep 08 13:18:56.381060 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.381040    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 13:18:56.381488 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.381469    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 13:18:56.381656 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.381494    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7sv8r\""
Sep 08 13:18:56.381723 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.381541    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 13:18:56.384815 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.384792    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"
Sep 08 13:18:56.388656 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.388639    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-hdfc4\""
Sep 08 13:18:56.388936 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.388914    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 08 13:18:56.389473 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.389452    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 08 13:18:56.390107 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.390084    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 08 13:18:56.396393 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.396366    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"]
Sep 08 13:18:56.451409 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.451368    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5b281d14-753f-4ac3-a725-ce90bca29a25-node-exporter-textfile\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.451409 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.451413    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ce9ed2a-e752-42d2-9c83-88ee0c17532d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-df4nq\" (UID: \"8ce9ed2a-e752-42d2-9c83-88ee0c17532d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq"
Sep 08 13:18:56.451666 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.451436    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-fhrnx\" (UID: \"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"
Sep 08 13:18:56.451666 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.451521    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvv2n\" (UniqueName: \"kubernetes.io/projected/8ce9ed2a-e752-42d2-9c83-88ee0c17532d-kube-api-access-fvv2n\") pod \"openshift-state-metrics-657fd6c54-df4nq\" (UID: \"8ce9ed2a-e752-42d2-9c83-88ee0c17532d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq"
Sep 08 13:18:56.451666 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.451562    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-fhrnx\" (UID: \"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"
Sep 08 13:18:56.451666 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.451584    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmvps\" (UniqueName: \"kubernetes.io/projected/b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986-kube-api-access-wmvps\") pod \"kube-state-metrics-74cb57df7-fhrnx\" (UID: \"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"
Sep 08 13:18:56.451666 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.451608    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wczv5\" (UniqueName: \"kubernetes.io/projected/5b281d14-753f-4ac3-a725-ce90bca29a25-kube-api-access-wczv5\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.451666 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.451626    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5b281d14-753f-4ac3-a725-ce90bca29a25-root\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.451666 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.451646    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5b281d14-753f-4ac3-a725-ce90bca29a25-node-exporter-tls\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.452019 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.451726    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5b281d14-753f-4ac3-a725-ce90bca29a25-node-exporter-wtmp\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.452019 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.451780    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-fhrnx\" (UID: \"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"
Sep 08 13:18:56.452019 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.451821    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ce9ed2a-e752-42d2-9c83-88ee0c17532d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-df4nq\" (UID: \"8ce9ed2a-e752-42d2-9c83-88ee0c17532d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq"
Sep 08 13:18:56.452019 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.451847    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b281d14-753f-4ac3-a725-ce90bca29a25-metrics-client-ca\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.452019 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.451887    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b281d14-753f-4ac3-a725-ce90bca29a25-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.452019 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.451914    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ce9ed2a-e752-42d2-9c83-88ee0c17532d-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-df4nq\" (UID: \"8ce9ed2a-e752-42d2-9c83-88ee0c17532d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq"
Sep 08 13:18:56.452019 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.451952    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5b281d14-753f-4ac3-a725-ce90bca29a25-node-exporter-accelerators-collector-config\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.452019 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.452000    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-fhrnx\" (UID: \"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"
Sep 08 13:18:56.452476 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.452048    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b281d14-753f-4ac3-a725-ce90bca29a25-sys\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.452476 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.452081    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-fhrnx\" (UID: \"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"
Sep 08 13:18:56.552756 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.552705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b281d14-753f-4ac3-a725-ce90bca29a25-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.552756 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.552760    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ce9ed2a-e752-42d2-9c83-88ee0c17532d-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-df4nq\" (UID: \"8ce9ed2a-e752-42d2-9c83-88ee0c17532d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq"
Sep 08 13:18:56.553025 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.552865    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5b281d14-753f-4ac3-a725-ce90bca29a25-node-exporter-accelerators-collector-config\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.553025 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.552894    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-fhrnx\" (UID: \"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"
Sep 08 13:18:56.553025 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.552920    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b281d14-753f-4ac3-a725-ce90bca29a25-sys\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.553025 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.552949    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-fhrnx\" (UID: \"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"
Sep 08 13:18:56.553025 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.552980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5b281d14-753f-4ac3-a725-ce90bca29a25-node-exporter-textfile\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.553025 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.553009    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ce9ed2a-e752-42d2-9c83-88ee0c17532d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-df4nq\" (UID: \"8ce9ed2a-e752-42d2-9c83-88ee0c17532d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq"
Sep 08 13:18:56.553353 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.553038    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-fhrnx\" (UID: \"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"
Sep 08 13:18:56.553353 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.553070    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvv2n\" (UniqueName: \"kubernetes.io/projected/8ce9ed2a-e752-42d2-9c83-88ee0c17532d-kube-api-access-fvv2n\") pod \"openshift-state-metrics-657fd6c54-df4nq\" (UID: \"8ce9ed2a-e752-42d2-9c83-88ee0c17532d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq"
Sep 08 13:18:56.553353 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.553116    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-fhrnx\" (UID: \"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"
Sep 08 13:18:56.553353 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.553139    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmvps\" (UniqueName: \"kubernetes.io/projected/b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986-kube-api-access-wmvps\") pod \"kube-state-metrics-74cb57df7-fhrnx\" (UID: \"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"
Sep 08 13:18:56.553353 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.553175    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wczv5\" (UniqueName: \"kubernetes.io/projected/5b281d14-753f-4ac3-a725-ce90bca29a25-kube-api-access-wczv5\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.553353 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.553200    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5b281d14-753f-4ac3-a725-ce90bca29a25-root\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.553353 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.553221    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5b281d14-753f-4ac3-a725-ce90bca29a25-node-exporter-tls\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.553353 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.553262    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5b281d14-753f-4ac3-a725-ce90bca29a25-node-exporter-wtmp\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.553353 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.553279    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-fhrnx\" (UID: \"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"
Sep 08 13:18:56.553787 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.553546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5b281d14-753f-4ac3-a725-ce90bca29a25-node-exporter-accelerators-collector-config\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.553787 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.553313    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-fhrnx\" (UID: \"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"
Sep 08 13:18:56.553787 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.553625    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ce9ed2a-e752-42d2-9c83-88ee0c17532d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-df4nq\" (UID: \"8ce9ed2a-e752-42d2-9c83-88ee0c17532d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq"
Sep 08 13:18:56.553787 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.553652    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b281d14-753f-4ac3-a725-ce90bca29a25-metrics-client-ca\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.553787 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.553729    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5b281d14-753f-4ac3-a725-ce90bca29a25-root\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.554146 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.554122    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b281d14-753f-4ac3-a725-ce90bca29a25-metrics-client-ca\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.554253 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:56.554219    2561 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 08 13:18:56.554335 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:18:56.554276    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ce9ed2a-e752-42d2-9c83-88ee0c17532d-openshift-state-metrics-tls podName:8ce9ed2a-e752-42d2-9c83-88ee0c17532d nodeName:}" failed. No retries permitted until 2026-09-08 13:18:57.054258109 +0000 UTC m=+168.514902903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8ce9ed2a-e752-42d2-9c83-88ee0c17532d-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-df4nq" (UID: "8ce9ed2a-e752-42d2-9c83-88ee0c17532d") : secret "openshift-state-metrics-tls" not found
Sep 08 13:18:56.554439 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.554414    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-fhrnx\" (UID: \"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"
Sep 08 13:18:56.554615 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.554594    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b281d14-753f-4ac3-a725-ce90bca29a25-sys\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.554718 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.554614    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5b281d14-753f-4ac3-a725-ce90bca29a25-node-exporter-wtmp\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.554865 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.554843    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5b281d14-753f-4ac3-a725-ce90bca29a25-node-exporter-textfile\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.555093 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.555067    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ce9ed2a-e752-42d2-9c83-88ee0c17532d-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-df4nq\" (UID: \"8ce9ed2a-e752-42d2-9c83-88ee0c17532d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq"
Sep 08 13:18:56.555650 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.555626    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-fhrnx\" (UID: \"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"
Sep 08 13:18:56.556586 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.556508    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-fhrnx\" (UID: \"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"
Sep 08 13:18:56.557613 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.557566    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b281d14-753f-4ac3-a725-ce90bca29a25-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.557756 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.557727    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-fhrnx\" (UID: \"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"
Sep 08 13:18:56.558029 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.558005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5b281d14-753f-4ac3-a725-ce90bca29a25-node-exporter-tls\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.558421 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.558399    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ce9ed2a-e752-42d2-9c83-88ee0c17532d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-df4nq\" (UID: \"8ce9ed2a-e752-42d2-9c83-88ee0c17532d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq"
Sep 08 13:18:56.566508 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.566484    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmvps\" (UniqueName: \"kubernetes.io/projected/b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986-kube-api-access-wmvps\") pod \"kube-state-metrics-74cb57df7-fhrnx\" (UID: \"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"
Sep 08 13:18:56.566862 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.566841    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wczv5\" (UniqueName: \"kubernetes.io/projected/5b281d14-753f-4ac3-a725-ce90bca29a25-kube-api-access-wczv5\") pod \"node-exporter-q7vd2\" (UID: \"5b281d14-753f-4ac3-a725-ce90bca29a25\") " pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.567509 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.567485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvv2n\" (UniqueName: \"kubernetes.io/projected/8ce9ed2a-e752-42d2-9c83-88ee0c17532d-kube-api-access-fvv2n\") pod \"openshift-state-metrics-657fd6c54-df4nq\" (UID: \"8ce9ed2a-e752-42d2-9c83-88ee0c17532d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq"
Sep 08 13:18:56.691198 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.690911    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q7vd2"
Sep 08 13:18:56.699506 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.699477    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"
Sep 08 13:18:56.704010 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:18:56.703976    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b281d14_753f_4ac3_a725_ce90bca29a25.slice/crio-52dffd2ca5b1a205edf7c0301c4cd65b69b802978e0670982f8b9eaea53826ad WatchSource:0}: Error finding container 52dffd2ca5b1a205edf7c0301c4cd65b69b802978e0670982f8b9eaea53826ad: Status 404 returned error can't find the container with id 52dffd2ca5b1a205edf7c0301c4cd65b69b802978e0670982f8b9eaea53826ad
Sep 08 13:18:56.983221 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:56.983136    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx"]
Sep 08 13:18:56.986551 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:18:56.986523    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0cb6c04_2dc0_4a79_83f5_cb6fc6c5e986.slice/crio-5e7d9859f77ec6bcc5548c845be95a3340dd33cb19c29b0f444c49227e0b0255 WatchSource:0}: Error finding container 5e7d9859f77ec6bcc5548c845be95a3340dd33cb19c29b0f444c49227e0b0255: Status 404 returned error can't find the container with id 5e7d9859f77ec6bcc5548c845be95a3340dd33cb19c29b0f444c49227e0b0255
Sep 08 13:18:57.000189 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:57.000160    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:18:57.059675 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:57.059636    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ce9ed2a-e752-42d2-9c83-88ee0c17532d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-df4nq\" (UID: \"8ce9ed2a-e752-42d2-9c83-88ee0c17532d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq"
Sep 08 13:18:57.061998 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:57.061974    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ce9ed2a-e752-42d2-9c83-88ee0c17532d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-df4nq\" (UID: \"8ce9ed2a-e752-42d2-9c83-88ee0c17532d\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq"
Sep 08 13:18:57.269649 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:57.269558    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq"
Sep 08 13:18:57.548368 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:57.548341    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq"]
Sep 08 13:18:57.550833 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:18:57.550809    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce9ed2a_e752_42d2_9c83_88ee0c17532d.slice/crio-91a163c94431fdd30cce9c5e660b73ff050223ae0e22842cb8a4e0f87003e341 WatchSource:0}: Error finding container 91a163c94431fdd30cce9c5e660b73ff050223ae0e22842cb8a4e0f87003e341: Status 404 returned error can't find the container with id 91a163c94431fdd30cce9c5e660b73ff050223ae0e22842cb8a4e0f87003e341
Sep 08 13:18:57.580470 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:57.580435    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx" event={"ID":"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986","Type":"ContainerStarted","Data":"5e7d9859f77ec6bcc5548c845be95a3340dd33cb19c29b0f444c49227e0b0255"}
Sep 08 13:18:57.581729 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:57.581703    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q7vd2" event={"ID":"5b281d14-753f-4ac3-a725-ce90bca29a25","Type":"ContainerStarted","Data":"52dffd2ca5b1a205edf7c0301c4cd65b69b802978e0670982f8b9eaea53826ad"}
Sep 08 13:18:57.582894 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:57.582873    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq" event={"ID":"8ce9ed2a-e752-42d2-9c83-88ee0c17532d","Type":"ContainerStarted","Data":"91a163c94431fdd30cce9c5e660b73ff050223ae0e22842cb8a4e0f87003e341"}
Sep 08 13:18:58.588477 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:58.588380    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq" event={"ID":"8ce9ed2a-e752-42d2-9c83-88ee0c17532d","Type":"ContainerStarted","Data":"2f955fff517d62405d51db0722164f9ac551ceea8fbadc6f71bd445a754b87e8"}
Sep 08 13:18:58.588477 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:58.588453    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq" event={"ID":"8ce9ed2a-e752-42d2-9c83-88ee0c17532d","Type":"ContainerStarted","Data":"9c4966244c3cd7a4f03a6c431a0303d8fae395580f10c2c59b1b10ce60ceb7c0"}
Sep 08 13:18:58.589896 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:58.589868    2561 generic.go:356] "Generic (PLEG): container finished" podID="5b281d14-753f-4ac3-a725-ce90bca29a25" containerID="b8f6f72e1c2190c8d7f8a5ff8115e84fbe99b4ab95fe7aff4c9c8a7cc1e5df3e" exitCode=0
Sep 08 13:18:58.590009 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:58.589913    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q7vd2" event={"ID":"5b281d14-753f-4ac3-a725-ce90bca29a25","Type":"ContainerDied","Data":"b8f6f72e1c2190c8d7f8a5ff8115e84fbe99b4ab95fe7aff4c9c8a7cc1e5df3e"}
Sep 08 13:18:59.596685 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:59.596646    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq" event={"ID":"8ce9ed2a-e752-42d2-9c83-88ee0c17532d","Type":"ContainerStarted","Data":"335e06fc9b38fc95dddff69c30adefd42de8bc9c4cf5a7280aec7aba9c9d8da5"}
Sep 08 13:18:59.598833 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:59.598807    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q7vd2" event={"ID":"5b281d14-753f-4ac3-a725-ce90bca29a25","Type":"ContainerStarted","Data":"f13f49ea4e5923d84f01d7e6bd5ff68cfc8e26c1d60e73922f08b8e7ca6665b2"}
Sep 08 13:18:59.598939 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:59.598851    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q7vd2" event={"ID":"5b281d14-753f-4ac3-a725-ce90bca29a25","Type":"ContainerStarted","Data":"0b996aced91b8af43fbe6926affc846c1f1a8d4fa73348605c2f0e4892ce89b0"}
Sep 08 13:18:59.614488 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:59.614437    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-df4nq" podStartSLOduration=2.131488524 podStartE2EDuration="3.614421904s" totalImagesPullingTime="1.48293338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:59.613304243 +0000 UTC m=+171.073949048" watchObservedRunningTime="2026-09-08 13:18:59.614421904 +0000 UTC m=+171.075066712"
Sep 08 13:18:59.636327 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:18:59.636261    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q7vd2" podStartSLOduration=2.015084274 podStartE2EDuration="3.636245958s" totalImagesPullingTime="1.555815251s" totalInitContainerRuntime="65.346433ms" isStatefulPod=true podCreationTimestamp="2026-09-08 13:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:59.635258017 +0000 UTC m=+171.095902827" watchObservedRunningTime="2026-09-08 13:18:59.636245958 +0000 UTC m=+171.096890777"
Sep 08 13:19:01.162080 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.161831    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79bb9c6c85-q4htq"]
Sep 08 13:19:01.165226 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.165193    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.184124 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.184077    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79bb9c6c85-q4htq"]
Sep 08 13:19:01.295772 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.295726    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-oauth-serving-cert\") pod \"console-79bb9c6c85-q4htq\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") " pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.295772 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.295775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-service-ca\") pod \"console-79bb9c6c85-q4htq\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") " pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.295995 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.295807    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-trusted-ca-bundle\") pod \"console-79bb9c6c85-q4htq\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") " pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.295995 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.295838    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2qzv\" (UniqueName: \"kubernetes.io/projected/c2a776c8-db90-4b13-b9ae-86d3a649d209-kube-api-access-r2qzv\") pod \"console-79bb9c6c85-q4htq\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") " pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.295995 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.295872    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-console-config\") pod \"console-79bb9c6c85-q4htq\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") " pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.295995 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.295894    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2a776c8-db90-4b13-b9ae-86d3a649d209-console-oauth-config\") pod \"console-79bb9c6c85-q4htq\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") " pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.295995 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.295923    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2a776c8-db90-4b13-b9ae-86d3a649d209-console-serving-cert\") pod \"console-79bb9c6c85-q4htq\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") " pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.397401 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.397354    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-oauth-serving-cert\") pod \"console-79bb9c6c85-q4htq\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") " pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.397589 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.397410    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-service-ca\") pod \"console-79bb9c6c85-q4htq\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") " pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.397589 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.397443    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-trusted-ca-bundle\") pod \"console-79bb9c6c85-q4htq\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") " pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.397589 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.397474    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2qzv\" (UniqueName: \"kubernetes.io/projected/c2a776c8-db90-4b13-b9ae-86d3a649d209-kube-api-access-r2qzv\") pod \"console-79bb9c6c85-q4htq\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") " pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.397589 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.397500    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-console-config\") pod \"console-79bb9c6c85-q4htq\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") " pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.397589 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.397529    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2a776c8-db90-4b13-b9ae-86d3a649d209-console-oauth-config\") pod \"console-79bb9c6c85-q4htq\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") " pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.397589 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.397567    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2a776c8-db90-4b13-b9ae-86d3a649d209-console-serving-cert\") pod \"console-79bb9c6c85-q4htq\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") " pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.398247 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.398213    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-oauth-serving-cert\") pod \"console-79bb9c6c85-q4htq\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") " pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.398492 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.398467    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-console-config\") pod \"console-79bb9c6c85-q4htq\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") " pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.398829 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.398802    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-service-ca\") pod \"console-79bb9c6c85-q4htq\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") " pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.398966 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.398941    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-trusted-ca-bundle\") pod \"console-79bb9c6c85-q4htq\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") " pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.400648 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.400625    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2a776c8-db90-4b13-b9ae-86d3a649d209-console-serving-cert\") pod \"console-79bb9c6c85-q4htq\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") " pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.401178 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.401152    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2a776c8-db90-4b13-b9ae-86d3a649d209-console-oauth-config\") pod \"console-79bb9c6c85-q4htq\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") " pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.406551 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.406525    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2qzv\" (UniqueName: \"kubernetes.io/projected/c2a776c8-db90-4b13-b9ae-86d3a649d209-kube-api-access-r2qzv\") pod \"console-79bb9c6c85-q4htq\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") " pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:01.487929 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:01.487883    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:02.028673 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:02.028640    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79bb9c6c85-q4htq"]
Sep 08 13:19:02.033037 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:19:02.033004    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a776c8_db90_4b13_b9ae_86d3a649d209.slice/crio-9c8e47b88c9911cc919d511a706012f0184ad8f8951e423db27b6b901edfdd19 WatchSource:0}: Error finding container 9c8e47b88c9911cc919d511a706012f0184ad8f8951e423db27b6b901edfdd19: Status 404 returned error can't find the container with id 9c8e47b88c9911cc919d511a706012f0184ad8f8951e423db27b6b901edfdd19
Sep 08 13:19:02.612411 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:02.612362    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx" event={"ID":"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986","Type":"ContainerStarted","Data":"9709ba83a8730f4e6f28372161c78aadf59a54a099508289c09ff5257d974ac6"}
Sep 08 13:19:02.612889 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:02.612439    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx" event={"ID":"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986","Type":"ContainerStarted","Data":"c3ad6f19a7086908ad319cf89641797d612019a1f8141e2c8310ed0b5a37f1e2"}
Sep 08 13:19:02.612889 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:02.612465    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx" event={"ID":"b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986","Type":"ContainerStarted","Data":"68f431c43a8be32a65887cabca7f72ed1bfd86f0a4c800bca224a826c4a83666"}
Sep 08 13:19:02.613976 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:02.613942    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bb9c6c85-q4htq" event={"ID":"c2a776c8-db90-4b13-b9ae-86d3a649d209","Type":"ContainerStarted","Data":"84694c1b773ab08ddd5bf2dd93fa30dc62cddffa5802dafdecf42fad1284ac37"}
Sep 08 13:19:02.614086 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:02.613996    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bb9c6c85-q4htq" event={"ID":"c2a776c8-db90-4b13-b9ae-86d3a649d209","Type":"ContainerStarted","Data":"9c8e47b88c9911cc919d511a706012f0184ad8f8951e423db27b6b901edfdd19"}
Sep 08 13:19:02.666622 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:02.666424    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fhrnx" podStartSLOduration=1.851289736 podStartE2EDuration="6.666400691s" totalImagesPullingTime="4.815110955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:02.664580313 +0000 UTC m=+174.125225136" watchObservedRunningTime="2026-09-08 13:19:02.666400691 +0000 UTC m=+174.127045501"
Sep 08 13:19:03.570368 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:03.570339    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-242sh"
Sep 08 13:19:03.589726 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:03.589658    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79bb9c6c85-q4htq" podStartSLOduration=2.589636111 podStartE2EDuration="2.589636111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:02.724628059 +0000 UTC m=+174.185272869" watchObservedRunningTime="2026-09-08 13:19:03.589636111 +0000 UTC m=+175.050280922"
Sep 08 13:19:08.942372 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:08.942333    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79bb9c6c85-q4htq"]
Sep 08 13:19:11.488567 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:11.488527    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:16.670527 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:16.670491    2561 generic.go:356] "Generic (PLEG): container finished" podID="509d01ad-7242-4f0f-9897-f0763e1b7298" containerID="888aa94e18f3149e948c313c720e6adeb44732e0b32baec72135bcba23e730ba" exitCode=0
Sep 08 13:19:16.670899 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:16.670558    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9q5lg" event={"ID":"509d01ad-7242-4f0f-9897-f0763e1b7298","Type":"ContainerDied","Data":"888aa94e18f3149e948c313c720e6adeb44732e0b32baec72135bcba23e730ba"}
Sep 08 13:19:16.670899 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:16.670851    2561 scope.go:117] "RemoveContainer" containerID="888aa94e18f3149e948c313c720e6adeb44732e0b32baec72135bcba23e730ba"
Sep 08 13:19:17.676023 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:17.675989    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-9q5lg" event={"ID":"509d01ad-7242-4f0f-9897-f0763e1b7298","Type":"ContainerStarted","Data":"2305c6f404697f0b5f1a6f906ded851030c33e1d528c2ec04a919e28cf613f09"}
Sep 08 13:19:19.912303 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:19.912255    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cc4fbd768-kwxdc"]
Sep 08 13:19:21.698319 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:21.698270    2561 generic.go:356] "Generic (PLEG): container finished" podID="549823a1-8d62-40a7-8de0-62cff329c97d" containerID="77db2d638c3ea04aaf42421bea4a08f7afd6cb05b834e3b35333fa594e31554c" exitCode=0
Sep 08 13:19:21.698823 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:21.698318    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-cvwkr" event={"ID":"549823a1-8d62-40a7-8de0-62cff329c97d","Type":"ContainerDied","Data":"77db2d638c3ea04aaf42421bea4a08f7afd6cb05b834e3b35333fa594e31554c"}
Sep 08 13:19:21.698823 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:21.698617    2561 scope.go:117] "RemoveContainer" containerID="77db2d638c3ea04aaf42421bea4a08f7afd6cb05b834e3b35333fa594e31554c"
Sep 08 13:19:22.704343 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:22.704309    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-cvwkr" event={"ID":"549823a1-8d62-40a7-8de0-62cff329c97d","Type":"ContainerStarted","Data":"9f6321961da97dcbd84b70e07e42d61e37f7117de76e897cfc1f72a8248cf66e"}
Sep 08 13:19:33.962664 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:33.962618    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79bb9c6c85-q4htq" podUID="c2a776c8-db90-4b13-b9ae-86d3a649d209" containerName="console" containerID="cri-o://84694c1b773ab08ddd5bf2dd93fa30dc62cddffa5802dafdecf42fad1284ac37" gracePeriod=15
Sep 08 13:19:34.202807 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.202780    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:34.280336 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.280223    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2a776c8-db90-4b13-b9ae-86d3a649d209-console-serving-cert\") pod \"c2a776c8-db90-4b13-b9ae-86d3a649d209\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") "
Sep 08 13:19:34.280336 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.280266    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-oauth-serving-cert\") pod \"c2a776c8-db90-4b13-b9ae-86d3a649d209\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") "
Sep 08 13:19:34.280336 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.280306    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2qzv\" (UniqueName: \"kubernetes.io/projected/c2a776c8-db90-4b13-b9ae-86d3a649d209-kube-api-access-r2qzv\") pod \"c2a776c8-db90-4b13-b9ae-86d3a649d209\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") "
Sep 08 13:19:34.280336 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.280341    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2a776c8-db90-4b13-b9ae-86d3a649d209-console-oauth-config\") pod \"c2a776c8-db90-4b13-b9ae-86d3a649d209\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") "
Sep 08 13:19:34.280662 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.280380    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-console-config\") pod \"c2a776c8-db90-4b13-b9ae-86d3a649d209\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") "
Sep 08 13:19:34.280662 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.280415    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-service-ca\") pod \"c2a776c8-db90-4b13-b9ae-86d3a649d209\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") "
Sep 08 13:19:34.280662 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.280463    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-trusted-ca-bundle\") pod \"c2a776c8-db90-4b13-b9ae-86d3a649d209\" (UID: \"c2a776c8-db90-4b13-b9ae-86d3a649d209\") "
Sep 08 13:19:34.280814 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.280623    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c2a776c8-db90-4b13-b9ae-86d3a649d209" (UID: "c2a776c8-db90-4b13-b9ae-86d3a649d209"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:19:34.280814 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.280802    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-console-config" (OuterVolumeSpecName: "console-config") pod "c2a776c8-db90-4b13-b9ae-86d3a649d209" (UID: "c2a776c8-db90-4b13-b9ae-86d3a649d209"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:19:34.280920 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.280887    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-service-ca" (OuterVolumeSpecName: "service-ca") pod "c2a776c8-db90-4b13-b9ae-86d3a649d209" (UID: "c2a776c8-db90-4b13-b9ae-86d3a649d209"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:19:34.281017 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.280994    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c2a776c8-db90-4b13-b9ae-86d3a649d209" (UID: "c2a776c8-db90-4b13-b9ae-86d3a649d209"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:19:34.282722 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.282694    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2a776c8-db90-4b13-b9ae-86d3a649d209-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c2a776c8-db90-4b13-b9ae-86d3a649d209" (UID: "c2a776c8-db90-4b13-b9ae-86d3a649d209"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:19:34.282722 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.282712    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2a776c8-db90-4b13-b9ae-86d3a649d209-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c2a776c8-db90-4b13-b9ae-86d3a649d209" (UID: "c2a776c8-db90-4b13-b9ae-86d3a649d209"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:19:34.282883 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.282715    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a776c8-db90-4b13-b9ae-86d3a649d209-kube-api-access-r2qzv" (OuterVolumeSpecName: "kube-api-access-r2qzv") pod "c2a776c8-db90-4b13-b9ae-86d3a649d209" (UID: "c2a776c8-db90-4b13-b9ae-86d3a649d209"). InnerVolumeSpecName "kube-api-access-r2qzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:19:34.381526 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.381473    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-console-config\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:19:34.381526 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.381517    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-service-ca\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:19:34.381526 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.381527    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-trusted-ca-bundle\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:19:34.381526 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.381537    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2a776c8-db90-4b13-b9ae-86d3a649d209-console-serving-cert\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:19:34.381526 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.381547    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2a776c8-db90-4b13-b9ae-86d3a649d209-oauth-serving-cert\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:19:34.381826 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.381555    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2qzv\" (UniqueName: \"kubernetes.io/projected/c2a776c8-db90-4b13-b9ae-86d3a649d209-kube-api-access-r2qzv\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:19:34.381826 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.381564    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2a776c8-db90-4b13-b9ae-86d3a649d209-console-oauth-config\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:19:34.752174 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.752141    2561 generic.go:356] "Generic (PLEG): container finished" podID="c2a776c8-db90-4b13-b9ae-86d3a649d209" containerID="84694c1b773ab08ddd5bf2dd93fa30dc62cddffa5802dafdecf42fad1284ac37" exitCode=0
Sep 08 13:19:34.752355 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.752188    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bb9c6c85-q4htq"
Sep 08 13:19:34.752355 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.752204    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bb9c6c85-q4htq" event={"ID":"c2a776c8-db90-4b13-b9ae-86d3a649d209","Type":"ContainerDied","Data":"84694c1b773ab08ddd5bf2dd93fa30dc62cddffa5802dafdecf42fad1284ac37"}
Sep 08 13:19:34.752355 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.752273    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bb9c6c85-q4htq" event={"ID":"c2a776c8-db90-4b13-b9ae-86d3a649d209","Type":"ContainerDied","Data":"9c8e47b88c9911cc919d511a706012f0184ad8f8951e423db27b6b901edfdd19"}
Sep 08 13:19:34.752355 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.752309    2561 scope.go:117] "RemoveContainer" containerID="84694c1b773ab08ddd5bf2dd93fa30dc62cddffa5802dafdecf42fad1284ac37"
Sep 08 13:19:34.761837 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.761813    2561 scope.go:117] "RemoveContainer" containerID="84694c1b773ab08ddd5bf2dd93fa30dc62cddffa5802dafdecf42fad1284ac37"
Sep 08 13:19:34.762138 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:19:34.762119    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84694c1b773ab08ddd5bf2dd93fa30dc62cddffa5802dafdecf42fad1284ac37\": container with ID starting with 84694c1b773ab08ddd5bf2dd93fa30dc62cddffa5802dafdecf42fad1284ac37 not found: ID does not exist" containerID="84694c1b773ab08ddd5bf2dd93fa30dc62cddffa5802dafdecf42fad1284ac37"
Sep 08 13:19:34.762195 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.762146    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84694c1b773ab08ddd5bf2dd93fa30dc62cddffa5802dafdecf42fad1284ac37"} err="failed to get container status \"84694c1b773ab08ddd5bf2dd93fa30dc62cddffa5802dafdecf42fad1284ac37\": rpc error: code = NotFound desc = could not find container \"84694c1b773ab08ddd5bf2dd93fa30dc62cddffa5802dafdecf42fad1284ac37\": container with ID starting with 84694c1b773ab08ddd5bf2dd93fa30dc62cddffa5802dafdecf42fad1284ac37 not found: ID does not exist"
Sep 08 13:19:34.773362 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.773333    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79bb9c6c85-q4htq"]
Sep 08 13:19:34.777839 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:34.777818    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79bb9c6c85-q4htq"]
Sep 08 13:19:35.005229 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:35.005150    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2a776c8-db90-4b13-b9ae-86d3a649d209" path="/var/lib/kubelet/pods/c2a776c8-db90-4b13-b9ae-86d3a649d209/volumes"
Sep 08 13:19:44.937136 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:44.937071    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cc4fbd768-kwxdc" podUID="db680cae-3410-41eb-a095-432aee141598" containerName="console" containerID="cri-o://ab319bb240d50bd4b19fa907f33cb22e7d6dbc999a1566ab5154436f0baef0d8" gracePeriod=15
Sep 08 13:19:45.183358 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.183331    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:19:45.272739 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.272629    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db680cae-3410-41eb-a095-432aee141598-console-oauth-config\") pod \"db680cae-3410-41eb-a095-432aee141598\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") "
Sep 08 13:19:45.272739 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.272691    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-oauth-serving-cert\") pod \"db680cae-3410-41eb-a095-432aee141598\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") "
Sep 08 13:19:45.272739 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.272713    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-console-config\") pod \"db680cae-3410-41eb-a095-432aee141598\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") "
Sep 08 13:19:45.272739 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.272746    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lb6mq\" (UniqueName: \"kubernetes.io/projected/db680cae-3410-41eb-a095-432aee141598-kube-api-access-lb6mq\") pod \"db680cae-3410-41eb-a095-432aee141598\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") "
Sep 08 13:19:45.273074 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.272769    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-trusted-ca-bundle\") pod \"db680cae-3410-41eb-a095-432aee141598\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") "
Sep 08 13:19:45.273074 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.272801    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db680cae-3410-41eb-a095-432aee141598-console-serving-cert\") pod \"db680cae-3410-41eb-a095-432aee141598\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") "
Sep 08 13:19:45.273074 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.272849    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-service-ca\") pod \"db680cae-3410-41eb-a095-432aee141598\" (UID: \"db680cae-3410-41eb-a095-432aee141598\") "
Sep 08 13:19:45.273231 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.273128    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "db680cae-3410-41eb-a095-432aee141598" (UID: "db680cae-3410-41eb-a095-432aee141598"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:19:45.273313 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.273241    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-console-config" (OuterVolumeSpecName: "console-config") pod "db680cae-3410-41eb-a095-432aee141598" (UID: "db680cae-3410-41eb-a095-432aee141598"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:19:45.273313 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.273274    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "db680cae-3410-41eb-a095-432aee141598" (UID: "db680cae-3410-41eb-a095-432aee141598"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:19:45.273417 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.273361    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-service-ca" (OuterVolumeSpecName: "service-ca") pod "db680cae-3410-41eb-a095-432aee141598" (UID: "db680cae-3410-41eb-a095-432aee141598"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:19:45.274984 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.274955    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db680cae-3410-41eb-a095-432aee141598-kube-api-access-lb6mq" (OuterVolumeSpecName: "kube-api-access-lb6mq") pod "db680cae-3410-41eb-a095-432aee141598" (UID: "db680cae-3410-41eb-a095-432aee141598"). InnerVolumeSpecName "kube-api-access-lb6mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:19:45.275329 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.275305    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db680cae-3410-41eb-a095-432aee141598-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "db680cae-3410-41eb-a095-432aee141598" (UID: "db680cae-3410-41eb-a095-432aee141598"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:19:45.275329 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.275317    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db680cae-3410-41eb-a095-432aee141598-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "db680cae-3410-41eb-a095-432aee141598" (UID: "db680cae-3410-41eb-a095-432aee141598"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:19:45.374271 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.374219    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-oauth-serving-cert\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:19:45.374271 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.374261    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-console-config\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:19:45.374271 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.374271    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb6mq\" (UniqueName: \"kubernetes.io/projected/db680cae-3410-41eb-a095-432aee141598-kube-api-access-lb6mq\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:19:45.374271 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.374280    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-trusted-ca-bundle\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:19:45.374271 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.374308    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db680cae-3410-41eb-a095-432aee141598-console-serving-cert\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:19:45.374271 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.374317    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db680cae-3410-41eb-a095-432aee141598-service-ca\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:19:45.374616 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.374326    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db680cae-3410-41eb-a095-432aee141598-console-oauth-config\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:19:45.796124 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.796094    2561 generic.go:356] "Generic (PLEG): container finished" podID="db680cae-3410-41eb-a095-432aee141598" containerID="ab319bb240d50bd4b19fa907f33cb22e7d6dbc999a1566ab5154436f0baef0d8" exitCode=0
Sep 08 13:19:45.796310 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.796138    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc4fbd768-kwxdc"
Sep 08 13:19:45.796310 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.796140    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc4fbd768-kwxdc" event={"ID":"db680cae-3410-41eb-a095-432aee141598","Type":"ContainerDied","Data":"ab319bb240d50bd4b19fa907f33cb22e7d6dbc999a1566ab5154436f0baef0d8"}
Sep 08 13:19:45.796310 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.796239    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc4fbd768-kwxdc" event={"ID":"db680cae-3410-41eb-a095-432aee141598","Type":"ContainerDied","Data":"be3bb7d645b75d2e60cd0388e816d2d95a04925a8881de87ac5133e719675a7c"}
Sep 08 13:19:45.796310 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.796260    2561 scope.go:117] "RemoveContainer" containerID="ab319bb240d50bd4b19fa907f33cb22e7d6dbc999a1566ab5154436f0baef0d8"
Sep 08 13:19:45.804945 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.804927    2561 scope.go:117] "RemoveContainer" containerID="ab319bb240d50bd4b19fa907f33cb22e7d6dbc999a1566ab5154436f0baef0d8"
Sep 08 13:19:45.805224 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:19:45.805206    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab319bb240d50bd4b19fa907f33cb22e7d6dbc999a1566ab5154436f0baef0d8\": container with ID starting with ab319bb240d50bd4b19fa907f33cb22e7d6dbc999a1566ab5154436f0baef0d8 not found: ID does not exist" containerID="ab319bb240d50bd4b19fa907f33cb22e7d6dbc999a1566ab5154436f0baef0d8"
Sep 08 13:19:45.805282 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.805235    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab319bb240d50bd4b19fa907f33cb22e7d6dbc999a1566ab5154436f0baef0d8"} err="failed to get container status \"ab319bb240d50bd4b19fa907f33cb22e7d6dbc999a1566ab5154436f0baef0d8\": rpc error: code = NotFound desc = could not find container \"ab319bb240d50bd4b19fa907f33cb22e7d6dbc999a1566ab5154436f0baef0d8\": container with ID starting with ab319bb240d50bd4b19fa907f33cb22e7d6dbc999a1566ab5154436f0baef0d8 not found: ID does not exist"
Sep 08 13:19:45.818420 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.818379    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cc4fbd768-kwxdc"]
Sep 08 13:19:45.822401 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:45.822376    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cc4fbd768-kwxdc"]
Sep 08 13:19:47.005591 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:19:47.005561    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db680cae-3410-41eb-a095-432aee141598" path="/var/lib/kubelet/pods/db680cae-3410-41eb-a095-432aee141598/volumes"
Sep 08 13:20:20.772015 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:20:20.771969    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs\") pod \"network-metrics-daemon-gsgmd\" (UID: \"246d684e-5ff4-49bb-b39c-660e46385842\") " pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:20:20.774511 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:20:20.774337    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/246d684e-5ff4-49bb-b39c-660e46385842-metrics-certs\") pod \"network-metrics-daemon-gsgmd\" (UID: \"246d684e-5ff4-49bb-b39c-660e46385842\") " pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:20:21.003331 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:20:21.003275    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4rwvw\""
Sep 08 13:20:21.011582 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:20:21.011556    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gsgmd"
Sep 08 13:20:21.291046 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:20:21.290965    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gsgmd"]
Sep 08 13:20:21.294767 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:20:21.294738    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246d684e_5ff4_49bb_b39c_660e46385842.slice/crio-131c8c9b4785f022d4846a81b5d1cd466403779370e5c9785ca7c5bf328390b8 WatchSource:0}: Error finding container 131c8c9b4785f022d4846a81b5d1cd466403779370e5c9785ca7c5bf328390b8: Status 404 returned error can't find the container with id 131c8c9b4785f022d4846a81b5d1cd466403779370e5c9785ca7c5bf328390b8
Sep 08 13:20:21.930368 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:20:21.930323    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gsgmd" event={"ID":"246d684e-5ff4-49bb-b39c-660e46385842","Type":"ContainerStarted","Data":"131c8c9b4785f022d4846a81b5d1cd466403779370e5c9785ca7c5bf328390b8"}
Sep 08 13:20:22.936240 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:20:22.936204    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gsgmd" event={"ID":"246d684e-5ff4-49bb-b39c-660e46385842","Type":"ContainerStarted","Data":"602dd62bbf3b3394b2d4fe872f25e786475da63c6e028622060879c4598a963b"}
Sep 08 13:20:22.936641 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:20:22.936257    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gsgmd" event={"ID":"246d684e-5ff4-49bb-b39c-660e46385842","Type":"ContainerStarted","Data":"e1b072b4897f763565b73d32cfed95c4fcbf16c0ee1b8877d7741926a61ec84c"}
Sep 08 13:20:22.959199 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:20:22.959144    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gsgmd" podStartSLOduration=252.973426752 podStartE2EDuration="4m13.959127243s" totalImagesPullingTime="985.700491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:20:22.958448557 +0000 UTC m=+254.419093374" watchObservedRunningTime="2026-09-08 13:20:22.959127243 +0000 UTC m=+254.419772052"
Sep 08 13:21:08.914743 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:21:08.914711    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 13:21:08.917047 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:21:08.917019    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 13:21:08.927623 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:21:08.927587    2561 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 13:22:08.287167 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.287063    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w"]
Sep 08 13:22:08.287611 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.287365    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2a776c8-db90-4b13-b9ae-86d3a649d209" containerName="console"
Sep 08 13:22:08.287611 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.287376    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2a776c8-db90-4b13-b9ae-86d3a649d209" containerName="console"
Sep 08 13:22:08.287611 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.287384    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db680cae-3410-41eb-a095-432aee141598" containerName="console"
Sep 08 13:22:08.287611 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.287391    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="db680cae-3410-41eb-a095-432aee141598" containerName="console"
Sep 08 13:22:08.287611 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.287431    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="db680cae-3410-41eb-a095-432aee141598" containerName="console"
Sep 08 13:22:08.287611 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.287442    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2a776c8-db90-4b13-b9ae-86d3a649d209" containerName="console"
Sep 08 13:22:08.290149 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.290131    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w"
Sep 08 13:22:08.292559 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.292534    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lwbc7\""
Sep 08 13:22:08.292686 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.292583    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 08 13:22:08.293178 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.293162    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 08 13:22:08.300612 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.300587    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w"]
Sep 08 13:22:08.325482 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.325450    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f18bb3d7-d9f5-4dc1-9995-0808d4099887-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w\" (UID: \"f18bb3d7-d9f5-4dc1-9995-0808d4099887\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w"
Sep 08 13:22:08.325674 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.325486    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfc5z\" (UniqueName: \"kubernetes.io/projected/f18bb3d7-d9f5-4dc1-9995-0808d4099887-kube-api-access-hfc5z\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w\" (UID: \"f18bb3d7-d9f5-4dc1-9995-0808d4099887\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w"
Sep 08 13:22:08.325674 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.325581    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f18bb3d7-d9f5-4dc1-9995-0808d4099887-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w\" (UID: \"f18bb3d7-d9f5-4dc1-9995-0808d4099887\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w"
Sep 08 13:22:08.426594 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.426556    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f18bb3d7-d9f5-4dc1-9995-0808d4099887-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w\" (UID: \"f18bb3d7-d9f5-4dc1-9995-0808d4099887\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w"
Sep 08 13:22:08.426738 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.426597    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfc5z\" (UniqueName: \"kubernetes.io/projected/f18bb3d7-d9f5-4dc1-9995-0808d4099887-kube-api-access-hfc5z\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w\" (UID: \"f18bb3d7-d9f5-4dc1-9995-0808d4099887\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w"
Sep 08 13:22:08.426738 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.426647    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f18bb3d7-d9f5-4dc1-9995-0808d4099887-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w\" (UID: \"f18bb3d7-d9f5-4dc1-9995-0808d4099887\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w"
Sep 08 13:22:08.426969 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.426945    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f18bb3d7-d9f5-4dc1-9995-0808d4099887-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w\" (UID: \"f18bb3d7-d9f5-4dc1-9995-0808d4099887\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w"
Sep 08 13:22:08.427038 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.426992    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f18bb3d7-d9f5-4dc1-9995-0808d4099887-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w\" (UID: \"f18bb3d7-d9f5-4dc1-9995-0808d4099887\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w"
Sep 08 13:22:08.435200 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.435170    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfc5z\" (UniqueName: \"kubernetes.io/projected/f18bb3d7-d9f5-4dc1-9995-0808d4099887-kube-api-access-hfc5z\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w\" (UID: \"f18bb3d7-d9f5-4dc1-9995-0808d4099887\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w"
Sep 08 13:22:08.599440 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.599334    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w"
Sep 08 13:22:08.870744 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.870670    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w"]
Sep 08 13:22:08.874298 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:22:08.874249    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf18bb3d7_d9f5_4dc1_9995_0808d4099887.slice/crio-e54c1e944b6dd2b8bb962a83bd65b4a192e6f13ab4f0e3770cdc3eb9a80857c0 WatchSource:0}: Error finding container e54c1e944b6dd2b8bb962a83bd65b4a192e6f13ab4f0e3770cdc3eb9a80857c0: Status 404 returned error can't find the container with id e54c1e944b6dd2b8bb962a83bd65b4a192e6f13ab4f0e3770cdc3eb9a80857c0
Sep 08 13:22:08.876423 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:08.876407    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:22:09.332211 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:09.332169    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w" event={"ID":"f18bb3d7-d9f5-4dc1-9995-0808d4099887","Type":"ContainerStarted","Data":"e54c1e944b6dd2b8bb962a83bd65b4a192e6f13ab4f0e3770cdc3eb9a80857c0"}
Sep 08 13:22:14.356554 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:14.356518    2561 generic.go:356] "Generic (PLEG): container finished" podID="f18bb3d7-d9f5-4dc1-9995-0808d4099887" containerID="53b5b0e10bc914637c1b3bf65187f274890bc18c98865550a6a7162391d81215" exitCode=0
Sep 08 13:22:14.357005 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:14.356612    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w" event={"ID":"f18bb3d7-d9f5-4dc1-9995-0808d4099887","Type":"ContainerDied","Data":"53b5b0e10bc914637c1b3bf65187f274890bc18c98865550a6a7162391d81215"}
Sep 08 13:22:17.370957 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:17.370923    2561 generic.go:356] "Generic (PLEG): container finished" podID="f18bb3d7-d9f5-4dc1-9995-0808d4099887" containerID="d4abd4d82f59f76efdefc82b44c661fff9735c5c7da539f548694a5d4b3592a2" exitCode=0
Sep 08 13:22:17.371457 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:17.370995    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w" event={"ID":"f18bb3d7-d9f5-4dc1-9995-0808d4099887","Type":"ContainerDied","Data":"d4abd4d82f59f76efdefc82b44c661fff9735c5c7da539f548694a5d4b3592a2"}
Sep 08 13:22:24.401927 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:24.401834    2561 generic.go:356] "Generic (PLEG): container finished" podID="f18bb3d7-d9f5-4dc1-9995-0808d4099887" containerID="d9f773a07946fe8254be7fad0d48d0bdd7aed5877a5f25206a86d43c30c5118b" exitCode=0
Sep 08 13:22:24.401927 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:24.401911    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w" event={"ID":"f18bb3d7-d9f5-4dc1-9995-0808d4099887","Type":"ContainerDied","Data":"d9f773a07946fe8254be7fad0d48d0bdd7aed5877a5f25206a86d43c30c5118b"}
Sep 08 13:22:25.533634 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:25.533609    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w"
Sep 08 13:22:25.671317 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:25.671208    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f18bb3d7-d9f5-4dc1-9995-0808d4099887-util\") pod \"f18bb3d7-d9f5-4dc1-9995-0808d4099887\" (UID: \"f18bb3d7-d9f5-4dc1-9995-0808d4099887\") "
Sep 08 13:22:25.671317 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:25.671262    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfc5z\" (UniqueName: \"kubernetes.io/projected/f18bb3d7-d9f5-4dc1-9995-0808d4099887-kube-api-access-hfc5z\") pod \"f18bb3d7-d9f5-4dc1-9995-0808d4099887\" (UID: \"f18bb3d7-d9f5-4dc1-9995-0808d4099887\") "
Sep 08 13:22:25.671317 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:25.671316    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f18bb3d7-d9f5-4dc1-9995-0808d4099887-bundle\") pod \"f18bb3d7-d9f5-4dc1-9995-0808d4099887\" (UID: \"f18bb3d7-d9f5-4dc1-9995-0808d4099887\") "
Sep 08 13:22:25.672040 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:25.672015    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18bb3d7-d9f5-4dc1-9995-0808d4099887-bundle" (OuterVolumeSpecName: "bundle") pod "f18bb3d7-d9f5-4dc1-9995-0808d4099887" (UID: "f18bb3d7-d9f5-4dc1-9995-0808d4099887"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:22:25.673627 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:25.673605    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f18bb3d7-d9f5-4dc1-9995-0808d4099887-kube-api-access-hfc5z" (OuterVolumeSpecName: "kube-api-access-hfc5z") pod "f18bb3d7-d9f5-4dc1-9995-0808d4099887" (UID: "f18bb3d7-d9f5-4dc1-9995-0808d4099887"). InnerVolumeSpecName "kube-api-access-hfc5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:22:25.675096 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:25.675069    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18bb3d7-d9f5-4dc1-9995-0808d4099887-util" (OuterVolumeSpecName: "util") pod "f18bb3d7-d9f5-4dc1-9995-0808d4099887" (UID: "f18bb3d7-d9f5-4dc1-9995-0808d4099887"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:22:25.772153 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:25.772116    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f18bb3d7-d9f5-4dc1-9995-0808d4099887-util\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:22:25.772153 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:25.772149    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfc5z\" (UniqueName: \"kubernetes.io/projected/f18bb3d7-d9f5-4dc1-9995-0808d4099887-kube-api-access-hfc5z\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:22:25.772153 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:25.772159    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f18bb3d7-d9f5-4dc1-9995-0808d4099887-bundle\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:22:26.411227 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:26.411199    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w"
Sep 08 13:22:26.411227 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:26.411206    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a5x54w" event={"ID":"f18bb3d7-d9f5-4dc1-9995-0808d4099887","Type":"ContainerDied","Data":"e54c1e944b6dd2b8bb962a83bd65b4a192e6f13ab4f0e3770cdc3eb9a80857c0"}
Sep 08 13:22:26.411444 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:26.411252    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e54c1e944b6dd2b8bb962a83bd65b4a192e6f13ab4f0e3770cdc3eb9a80857c0"
Sep 08 13:22:36.445880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.445844    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-t6v76"]
Sep 08 13:22:36.446264 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.446125    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f18bb3d7-d9f5-4dc1-9995-0808d4099887" containerName="pull"
Sep 08 13:22:36.446264 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.446136    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f18bb3d7-d9f5-4dc1-9995-0808d4099887" containerName="pull"
Sep 08 13:22:36.446264 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.446144    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f18bb3d7-d9f5-4dc1-9995-0808d4099887" containerName="extract"
Sep 08 13:22:36.446264 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.446150    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f18bb3d7-d9f5-4dc1-9995-0808d4099887" containerName="extract"
Sep 08 13:22:36.446264 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.446158    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f18bb3d7-d9f5-4dc1-9995-0808d4099887" containerName="util"
Sep 08 13:22:36.446264 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.446164    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f18bb3d7-d9f5-4dc1-9995-0808d4099887" containerName="util"
Sep 08 13:22:36.446264 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.446220    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f18bb3d7-d9f5-4dc1-9995-0808d4099887" containerName="extract"
Sep 08 13:22:36.467739 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.467707    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-t6v76"]
Sep 08 13:22:36.467887 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.467809    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-t6v76"
Sep 08 13:22:36.470243 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.470219    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 08 13:22:36.470946 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.470926    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 08 13:22:36.471058 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.470983    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 08 13:22:36.471058 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.471034    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 08 13:22:36.471183 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.470983    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 08 13:22:36.471257 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.471243    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-nsq54\""
Sep 08 13:22:36.567038 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.566989    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shzpc\" (UniqueName: \"kubernetes.io/projected/69268ef5-45c3-45aa-b8e7-8683047c0b56-kube-api-access-shzpc\") pod \"keda-metrics-apiserver-77857b7b98-t6v76\" (UID: \"69268ef5-45c3-45aa-b8e7-8683047c0b56\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-t6v76"
Sep 08 13:22:36.567226 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.567077    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/69268ef5-45c3-45aa-b8e7-8683047c0b56-certificates\") pod \"keda-metrics-apiserver-77857b7b98-t6v76\" (UID: \"69268ef5-45c3-45aa-b8e7-8683047c0b56\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-t6v76"
Sep 08 13:22:36.567226 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.567133    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/69268ef5-45c3-45aa-b8e7-8683047c0b56-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-t6v76\" (UID: \"69268ef5-45c3-45aa-b8e7-8683047c0b56\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-t6v76"
Sep 08 13:22:36.667640 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.667602    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shzpc\" (UniqueName: \"kubernetes.io/projected/69268ef5-45c3-45aa-b8e7-8683047c0b56-kube-api-access-shzpc\") pod \"keda-metrics-apiserver-77857b7b98-t6v76\" (UID: \"69268ef5-45c3-45aa-b8e7-8683047c0b56\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-t6v76"
Sep 08 13:22:36.667836 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.667653    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/69268ef5-45c3-45aa-b8e7-8683047c0b56-certificates\") pod \"keda-metrics-apiserver-77857b7b98-t6v76\" (UID: \"69268ef5-45c3-45aa-b8e7-8683047c0b56\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-t6v76"
Sep 08 13:22:36.667836 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.667684    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/69268ef5-45c3-45aa-b8e7-8683047c0b56-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-t6v76\" (UID: \"69268ef5-45c3-45aa-b8e7-8683047c0b56\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-t6v76"
Sep 08 13:22:36.667836 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:22:36.667807    2561 secret.go:281] references non-existent secret key: tls.crt
Sep 08 13:22:36.667836 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:22:36.667828    2561 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 08 13:22:36.668030 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:22:36.667844    2561 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 08 13:22:36.668030 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:22:36.667861    2561 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-t6v76: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 08 13:22:36.668030 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:22:36.667926    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69268ef5-45c3-45aa-b8e7-8683047c0b56-certificates podName:69268ef5-45c3-45aa-b8e7-8683047c0b56 nodeName:}" failed. No retries permitted until 2026-09-08 13:22:37.167910347 +0000 UTC m=+388.628555135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/69268ef5-45c3-45aa-b8e7-8683047c0b56-certificates") pod "keda-metrics-apiserver-77857b7b98-t6v76" (UID: "69268ef5-45c3-45aa-b8e7-8683047c0b56") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 08 13:22:36.668135 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.668044    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/69268ef5-45c3-45aa-b8e7-8683047c0b56-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-t6v76\" (UID: \"69268ef5-45c3-45aa-b8e7-8683047c0b56\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-t6v76"
Sep 08 13:22:36.677945 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:36.677919    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shzpc\" (UniqueName: \"kubernetes.io/projected/69268ef5-45c3-45aa-b8e7-8683047c0b56-kube-api-access-shzpc\") pod \"keda-metrics-apiserver-77857b7b98-t6v76\" (UID: \"69268ef5-45c3-45aa-b8e7-8683047c0b56\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-t6v76"
Sep 08 13:22:37.171708 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:37.171669    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/69268ef5-45c3-45aa-b8e7-8683047c0b56-certificates\") pod \"keda-metrics-apiserver-77857b7b98-t6v76\" (UID: \"69268ef5-45c3-45aa-b8e7-8683047c0b56\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-t6v76"
Sep 08 13:22:37.174250 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:37.174223    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/69268ef5-45c3-45aa-b8e7-8683047c0b56-certificates\") pod \"keda-metrics-apiserver-77857b7b98-t6v76\" (UID: \"69268ef5-45c3-45aa-b8e7-8683047c0b56\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-t6v76"
Sep 08 13:22:37.380745 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:37.380704    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-t6v76"
Sep 08 13:22:37.658049 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:37.657974    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-t6v76"]
Sep 08 13:22:37.660194 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:22:37.660162    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69268ef5_45c3_45aa_b8e7_8683047c0b56.slice/crio-089d07cf2433d80b9814888066ce5d5023827d35e7aeddb22f712ef06703bac0 WatchSource:0}: Error finding container 089d07cf2433d80b9814888066ce5d5023827d35e7aeddb22f712ef06703bac0: Status 404 returned error can't find the container with id 089d07cf2433d80b9814888066ce5d5023827d35e7aeddb22f712ef06703bac0
Sep 08 13:22:38.462445 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:38.462410    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-t6v76" event={"ID":"69268ef5-45c3-45aa-b8e7-8683047c0b56","Type":"ContainerStarted","Data":"089d07cf2433d80b9814888066ce5d5023827d35e7aeddb22f712ef06703bac0"}
Sep 08 13:22:40.472162 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:40.472118    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-t6v76" event={"ID":"69268ef5-45c3-45aa-b8e7-8683047c0b56","Type":"ContainerStarted","Data":"1c284078285a2a26b87993b3be41b38054cb4d84985bd3f841a95e5e3679c5fc"}
Sep 08 13:22:40.472685 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:40.472259    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-t6v76"
Sep 08 13:22:40.493353 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:40.493299    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-t6v76" podStartSLOduration=1.809171009 podStartE2EDuration="4.493266177s" totalImagesPullingTime="2.684095168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:22:40.492735165 +0000 UTC m=+391.953379975" watchObservedRunningTime="2026-09-08 13:22:40.493266177 +0000 UTC m=+391.953911344"
Sep 08 13:22:51.481042 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:22:51.481012    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-t6v76"
Sep 08 13:23:28.700345 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:28.700283    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg"]
Sep 08 13:23:28.705049 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:28.705030    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg"
Sep 08 13:23:28.707301 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:28.707259    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 08 13:23:28.707877 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:28.707859    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lwbc7\""
Sep 08 13:23:28.707969 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:28.707878    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 08 13:23:28.712628 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:28.712608    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg"]
Sep 08 13:23:28.775222 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:28.775179    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdq28\" (UniqueName: \"kubernetes.io/projected/36079655-0454-41e3-a7eb-451bc344b1f3-kube-api-access-hdq28\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg\" (UID: \"36079655-0454-41e3-a7eb-451bc344b1f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg"
Sep 08 13:23:28.775443 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:28.775240    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36079655-0454-41e3-a7eb-451bc344b1f3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg\" (UID: \"36079655-0454-41e3-a7eb-451bc344b1f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg"
Sep 08 13:23:28.775443 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:28.775309    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36079655-0454-41e3-a7eb-451bc344b1f3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg\" (UID: \"36079655-0454-41e3-a7eb-451bc344b1f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg"
Sep 08 13:23:28.876492 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:28.876437    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdq28\" (UniqueName: \"kubernetes.io/projected/36079655-0454-41e3-a7eb-451bc344b1f3-kube-api-access-hdq28\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg\" (UID: \"36079655-0454-41e3-a7eb-451bc344b1f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg"
Sep 08 13:23:28.876669 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:28.876509    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36079655-0454-41e3-a7eb-451bc344b1f3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg\" (UID: \"36079655-0454-41e3-a7eb-451bc344b1f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg"
Sep 08 13:23:28.876669 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:28.876568    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36079655-0454-41e3-a7eb-451bc344b1f3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg\" (UID: \"36079655-0454-41e3-a7eb-451bc344b1f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg"
Sep 08 13:23:28.876907 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:28.876882    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36079655-0454-41e3-a7eb-451bc344b1f3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg\" (UID: \"36079655-0454-41e3-a7eb-451bc344b1f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg"
Sep 08 13:23:28.876940 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:28.876926    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36079655-0454-41e3-a7eb-451bc344b1f3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg\" (UID: \"36079655-0454-41e3-a7eb-451bc344b1f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg"
Sep 08 13:23:28.886874 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:28.886851    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdq28\" (UniqueName: \"kubernetes.io/projected/36079655-0454-41e3-a7eb-451bc344b1f3-kube-api-access-hdq28\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg\" (UID: \"36079655-0454-41e3-a7eb-451bc344b1f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg"
Sep 08 13:23:29.014921 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:29.014834    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg"
Sep 08 13:23:29.292112 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:29.292087    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg"]
Sep 08 13:23:29.294683 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:23:29.294657    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36079655_0454_41e3_a7eb_451bc344b1f3.slice/crio-3351c30073f9445d643f70d2dc25f0824807a631eb7d4dc2e0ad7d2830c0aa16 WatchSource:0}: Error finding container 3351c30073f9445d643f70d2dc25f0824807a631eb7d4dc2e0ad7d2830c0aa16: Status 404 returned error can't find the container with id 3351c30073f9445d643f70d2dc25f0824807a631eb7d4dc2e0ad7d2830c0aa16
Sep 08 13:23:29.674031 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:29.673944    2561 generic.go:356] "Generic (PLEG): container finished" podID="36079655-0454-41e3-a7eb-451bc344b1f3" containerID="38d1c359bc50b8506d5b0da12c9a17c5ba6338c45ff2b1ea8a677f144e1ea27f" exitCode=0
Sep 08 13:23:29.674168 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:29.674026    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg" event={"ID":"36079655-0454-41e3-a7eb-451bc344b1f3","Type":"ContainerDied","Data":"38d1c359bc50b8506d5b0da12c9a17c5ba6338c45ff2b1ea8a677f144e1ea27f"}
Sep 08 13:23:29.674168 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:29.674082    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg" event={"ID":"36079655-0454-41e3-a7eb-451bc344b1f3","Type":"ContainerStarted","Data":"3351c30073f9445d643f70d2dc25f0824807a631eb7d4dc2e0ad7d2830c0aa16"}
Sep 08 13:23:31.685828 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:31.685797    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg" event={"ID":"36079655-0454-41e3-a7eb-451bc344b1f3","Type":"ContainerStarted","Data":"964b75a33f74ba60721a411f26131024943abdb06c6dffd0494f1f0ac6e5c83f"}
Sep 08 13:23:32.694670 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:32.694632    2561 generic.go:356] "Generic (PLEG): container finished" podID="36079655-0454-41e3-a7eb-451bc344b1f3" containerID="964b75a33f74ba60721a411f26131024943abdb06c6dffd0494f1f0ac6e5c83f" exitCode=0
Sep 08 13:23:32.695051 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:32.694713    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg" event={"ID":"36079655-0454-41e3-a7eb-451bc344b1f3","Type":"ContainerDied","Data":"964b75a33f74ba60721a411f26131024943abdb06c6dffd0494f1f0ac6e5c83f"}
Sep 08 13:23:33.700446 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:33.700410    2561 generic.go:356] "Generic (PLEG): container finished" podID="36079655-0454-41e3-a7eb-451bc344b1f3" containerID="e7747a34342411918f96da192104f18817aaf5dd78e9aaa62bd3f5dfe97d9020" exitCode=0
Sep 08 13:23:33.700446 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:33.700447    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg" event={"ID":"36079655-0454-41e3-a7eb-451bc344b1f3","Type":"ContainerDied","Data":"e7747a34342411918f96da192104f18817aaf5dd78e9aaa62bd3f5dfe97d9020"}
Sep 08 13:23:34.825733 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:34.825709    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg"
Sep 08 13:23:34.927267 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:34.927232    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36079655-0454-41e3-a7eb-451bc344b1f3-util\") pod \"36079655-0454-41e3-a7eb-451bc344b1f3\" (UID: \"36079655-0454-41e3-a7eb-451bc344b1f3\") "
Sep 08 13:23:34.927475 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:34.927308    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdq28\" (UniqueName: \"kubernetes.io/projected/36079655-0454-41e3-a7eb-451bc344b1f3-kube-api-access-hdq28\") pod \"36079655-0454-41e3-a7eb-451bc344b1f3\" (UID: \"36079655-0454-41e3-a7eb-451bc344b1f3\") "
Sep 08 13:23:34.927475 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:34.927338    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36079655-0454-41e3-a7eb-451bc344b1f3-bundle\") pod \"36079655-0454-41e3-a7eb-451bc344b1f3\" (UID: \"36079655-0454-41e3-a7eb-451bc344b1f3\") "
Sep 08 13:23:34.928100 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:34.928073    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36079655-0454-41e3-a7eb-451bc344b1f3-bundle" (OuterVolumeSpecName: "bundle") pod "36079655-0454-41e3-a7eb-451bc344b1f3" (UID: "36079655-0454-41e3-a7eb-451bc344b1f3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:23:34.929490 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:34.929466    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36079655-0454-41e3-a7eb-451bc344b1f3-kube-api-access-hdq28" (OuterVolumeSpecName: "kube-api-access-hdq28") pod "36079655-0454-41e3-a7eb-451bc344b1f3" (UID: "36079655-0454-41e3-a7eb-451bc344b1f3"). InnerVolumeSpecName "kube-api-access-hdq28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:23:34.932716 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:34.932675    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36079655-0454-41e3-a7eb-451bc344b1f3-util" (OuterVolumeSpecName: "util") pod "36079655-0454-41e3-a7eb-451bc344b1f3" (UID: "36079655-0454-41e3-a7eb-451bc344b1f3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:23:35.027977 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:35.027940    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36079655-0454-41e3-a7eb-451bc344b1f3-bundle\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:23:35.027977 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:35.027971    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36079655-0454-41e3-a7eb-451bc344b1f3-util\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:23:35.028157 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:35.027984    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdq28\" (UniqueName: \"kubernetes.io/projected/36079655-0454-41e3-a7eb-451bc344b1f3-kube-api-access-hdq28\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:23:35.710727 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:35.710700    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg"
Sep 08 13:23:35.710727 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:35.710719    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z4gzg" event={"ID":"36079655-0454-41e3-a7eb-451bc344b1f3","Type":"ContainerDied","Data":"3351c30073f9445d643f70d2dc25f0824807a631eb7d4dc2e0ad7d2830c0aa16"}
Sep 08 13:23:35.710936 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:35.710760    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3351c30073f9445d643f70d2dc25f0824807a631eb7d4dc2e0ad7d2830c0aa16"
Sep 08 13:23:40.745351 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.745314    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-577596d6f-fsg59"]
Sep 08 13:23:40.745722 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.745586    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36079655-0454-41e3-a7eb-451bc344b1f3" containerName="extract"
Sep 08 13:23:40.745722 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.745597    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="36079655-0454-41e3-a7eb-451bc344b1f3" containerName="extract"
Sep 08 13:23:40.745722 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.745612    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36079655-0454-41e3-a7eb-451bc344b1f3" containerName="pull"
Sep 08 13:23:40.745722 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.745619    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="36079655-0454-41e3-a7eb-451bc344b1f3" containerName="pull"
Sep 08 13:23:40.745722 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.745625    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36079655-0454-41e3-a7eb-451bc344b1f3" containerName="util"
Sep 08 13:23:40.745722 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.745631    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="36079655-0454-41e3-a7eb-451bc344b1f3" containerName="util"
Sep 08 13:23:40.745722 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.745686    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="36079655-0454-41e3-a7eb-451bc344b1f3" containerName="extract"
Sep 08 13:23:40.747725 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.747699    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:40.751439 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.751415    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 08 13:23:40.751560 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.751439    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 08 13:23:40.751560 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.751415    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 08 13:23:40.752517 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.752497    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 08 13:23:40.752517 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.752509    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 08 13:23:40.752680 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.752558    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 08 13:23:40.752680 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.752513    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 08 13:23:40.752680 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.752611    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kzhlx\""
Sep 08 13:23:40.760786 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.760764    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 08 13:23:40.762655 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.762634    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-577596d6f-fsg59"]
Sep 08 13:23:40.876625 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.876588    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s9k4\" (UniqueName: \"kubernetes.io/projected/af47a1c7-099c-4fb1-9f65-3beb22ed88ab-kube-api-access-7s9k4\") pod \"console-577596d6f-fsg59\" (UID: \"af47a1c7-099c-4fb1-9f65-3beb22ed88ab\") " pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:40.876790 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.876648    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af47a1c7-099c-4fb1-9f65-3beb22ed88ab-console-config\") pod \"console-577596d6f-fsg59\" (UID: \"af47a1c7-099c-4fb1-9f65-3beb22ed88ab\") " pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:40.876790 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.876722    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af47a1c7-099c-4fb1-9f65-3beb22ed88ab-console-oauth-config\") pod \"console-577596d6f-fsg59\" (UID: \"af47a1c7-099c-4fb1-9f65-3beb22ed88ab\") " pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:40.876790 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.876751    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af47a1c7-099c-4fb1-9f65-3beb22ed88ab-console-serving-cert\") pod \"console-577596d6f-fsg59\" (UID: \"af47a1c7-099c-4fb1-9f65-3beb22ed88ab\") " pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:40.876790 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.876775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af47a1c7-099c-4fb1-9f65-3beb22ed88ab-service-ca\") pod \"console-577596d6f-fsg59\" (UID: \"af47a1c7-099c-4fb1-9f65-3beb22ed88ab\") " pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:40.876927 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.876795    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af47a1c7-099c-4fb1-9f65-3beb22ed88ab-trusted-ca-bundle\") pod \"console-577596d6f-fsg59\" (UID: \"af47a1c7-099c-4fb1-9f65-3beb22ed88ab\") " pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:40.876927 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.876842    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af47a1c7-099c-4fb1-9f65-3beb22ed88ab-oauth-serving-cert\") pod \"console-577596d6f-fsg59\" (UID: \"af47a1c7-099c-4fb1-9f65-3beb22ed88ab\") " pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:40.978054 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.978018    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af47a1c7-099c-4fb1-9f65-3beb22ed88ab-service-ca\") pod \"console-577596d6f-fsg59\" (UID: \"af47a1c7-099c-4fb1-9f65-3beb22ed88ab\") " pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:40.978054 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.978058    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af47a1c7-099c-4fb1-9f65-3beb22ed88ab-trusted-ca-bundle\") pod \"console-577596d6f-fsg59\" (UID: \"af47a1c7-099c-4fb1-9f65-3beb22ed88ab\") " pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:40.978447 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.978084    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af47a1c7-099c-4fb1-9f65-3beb22ed88ab-oauth-serving-cert\") pod \"console-577596d6f-fsg59\" (UID: \"af47a1c7-099c-4fb1-9f65-3beb22ed88ab\") " pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:40.978447 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.978113    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s9k4\" (UniqueName: \"kubernetes.io/projected/af47a1c7-099c-4fb1-9f65-3beb22ed88ab-kube-api-access-7s9k4\") pod \"console-577596d6f-fsg59\" (UID: \"af47a1c7-099c-4fb1-9f65-3beb22ed88ab\") " pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:40.978447 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.978163    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af47a1c7-099c-4fb1-9f65-3beb22ed88ab-console-config\") pod \"console-577596d6f-fsg59\" (UID: \"af47a1c7-099c-4fb1-9f65-3beb22ed88ab\") " pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:40.978447 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.978222    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af47a1c7-099c-4fb1-9f65-3beb22ed88ab-console-oauth-config\") pod \"console-577596d6f-fsg59\" (UID: \"af47a1c7-099c-4fb1-9f65-3beb22ed88ab\") " pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:40.978447 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.978250    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af47a1c7-099c-4fb1-9f65-3beb22ed88ab-console-serving-cert\") pod \"console-577596d6f-fsg59\" (UID: \"af47a1c7-099c-4fb1-9f65-3beb22ed88ab\") " pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:40.979186 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.979149    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af47a1c7-099c-4fb1-9f65-3beb22ed88ab-oauth-serving-cert\") pod \"console-577596d6f-fsg59\" (UID: \"af47a1c7-099c-4fb1-9f65-3beb22ed88ab\") " pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:40.979339 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.979199    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af47a1c7-099c-4fb1-9f65-3beb22ed88ab-service-ca\") pod \"console-577596d6f-fsg59\" (UID: \"af47a1c7-099c-4fb1-9f65-3beb22ed88ab\") " pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:40.979477 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.979453    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af47a1c7-099c-4fb1-9f65-3beb22ed88ab-console-config\") pod \"console-577596d6f-fsg59\" (UID: \"af47a1c7-099c-4fb1-9f65-3beb22ed88ab\") " pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:40.979661 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.979637    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af47a1c7-099c-4fb1-9f65-3beb22ed88ab-trusted-ca-bundle\") pod \"console-577596d6f-fsg59\" (UID: \"af47a1c7-099c-4fb1-9f65-3beb22ed88ab\") " pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:40.981722 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.981693    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af47a1c7-099c-4fb1-9f65-3beb22ed88ab-console-serving-cert\") pod \"console-577596d6f-fsg59\" (UID: \"af47a1c7-099c-4fb1-9f65-3beb22ed88ab\") " pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:40.982124 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.982104    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af47a1c7-099c-4fb1-9f65-3beb22ed88ab-console-oauth-config\") pod \"console-577596d6f-fsg59\" (UID: \"af47a1c7-099c-4fb1-9f65-3beb22ed88ab\") " pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:40.989922 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:40.989899    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s9k4\" (UniqueName: \"kubernetes.io/projected/af47a1c7-099c-4fb1-9f65-3beb22ed88ab-kube-api-access-7s9k4\") pod \"console-577596d6f-fsg59\" (UID: \"af47a1c7-099c-4fb1-9f65-3beb22ed88ab\") " pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:41.058438 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:41.058344    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:41.354037 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:41.353953    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-577596d6f-fsg59"]
Sep 08 13:23:41.357226 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:23:41.357201    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf47a1c7_099c_4fb1_9f65_3beb22ed88ab.slice/crio-6d56ebcddde2ddd52c75f15000b3964ef8c46f7c52f2458653a110b51b82e06f WatchSource:0}: Error finding container 6d56ebcddde2ddd52c75f15000b3964ef8c46f7c52f2458653a110b51b82e06f: Status 404 returned error can't find the container with id 6d56ebcddde2ddd52c75f15000b3964ef8c46f7c52f2458653a110b51b82e06f
Sep 08 13:23:41.735654 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:41.735620    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-577596d6f-fsg59" event={"ID":"af47a1c7-099c-4fb1-9f65-3beb22ed88ab","Type":"ContainerStarted","Data":"97c2dd286000e41d7b3c245be3e4e78c4f1218628410865d7306e99a1113ed94"}
Sep 08 13:23:41.735831 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:41.735668    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-577596d6f-fsg59" event={"ID":"af47a1c7-099c-4fb1-9f65-3beb22ed88ab","Type":"ContainerStarted","Data":"6d56ebcddde2ddd52c75f15000b3964ef8c46f7c52f2458653a110b51b82e06f"}
Sep 08 13:23:41.769049 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:41.768997    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-577596d6f-fsg59" podStartSLOduration=1.7689803240000002 podStartE2EDuration="1.768980324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:23:41.767121954 +0000 UTC m=+453.227766795" watchObservedRunningTime="2026-09-08 13:23:41.768980324 +0000 UTC m=+453.229625134"
Sep 08 13:23:42.002826 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:42.002742    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hp86f"]
Sep 08 13:23:42.004989 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:42.004967    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hp86f"
Sep 08 13:23:42.012158 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:42.012136    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 08 13:23:42.012258 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:42.012197    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:23:42.012371 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:42.012354    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-jm5hd\""
Sep 08 13:23:42.036336 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:42.036302    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hp86f"]
Sep 08 13:23:42.089187 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:42.089140    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwx27\" (UniqueName: \"kubernetes.io/projected/1b6be9ca-4f36-4dba-b1c0-9bf4487d2280-kube-api-access-hwx27\") pod \"cert-manager-operator-controller-manager-684f79c9b9-hp86f\" (UID: \"1b6be9ca-4f36-4dba-b1c0-9bf4487d2280\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hp86f"
Sep 08 13:23:42.089356 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:42.089248    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b6be9ca-4f36-4dba-b1c0-9bf4487d2280-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-hp86f\" (UID: \"1b6be9ca-4f36-4dba-b1c0-9bf4487d2280\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hp86f"
Sep 08 13:23:42.189745 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:42.189707    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwx27\" (UniqueName: \"kubernetes.io/projected/1b6be9ca-4f36-4dba-b1c0-9bf4487d2280-kube-api-access-hwx27\") pod \"cert-manager-operator-controller-manager-684f79c9b9-hp86f\" (UID: \"1b6be9ca-4f36-4dba-b1c0-9bf4487d2280\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hp86f"
Sep 08 13:23:42.189911 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:42.189773    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b6be9ca-4f36-4dba-b1c0-9bf4487d2280-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-hp86f\" (UID: \"1b6be9ca-4f36-4dba-b1c0-9bf4487d2280\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hp86f"
Sep 08 13:23:42.190134 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:42.190118    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b6be9ca-4f36-4dba-b1c0-9bf4487d2280-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-hp86f\" (UID: \"1b6be9ca-4f36-4dba-b1c0-9bf4487d2280\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hp86f"
Sep 08 13:23:42.203692 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:42.203664    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwx27\" (UniqueName: \"kubernetes.io/projected/1b6be9ca-4f36-4dba-b1c0-9bf4487d2280-kube-api-access-hwx27\") pod \"cert-manager-operator-controller-manager-684f79c9b9-hp86f\" (UID: \"1b6be9ca-4f36-4dba-b1c0-9bf4487d2280\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hp86f"
Sep 08 13:23:42.314887 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:42.314804    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hp86f"
Sep 08 13:23:42.593650 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:42.593622    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hp86f"]
Sep 08 13:23:42.596601 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:23:42.596572    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6be9ca_4f36_4dba_b1c0_9bf4487d2280.slice/crio-e056739b11dd57f4315f6486b888e0fa128eac9d43167e7563f1333d351a0459 WatchSource:0}: Error finding container e056739b11dd57f4315f6486b888e0fa128eac9d43167e7563f1333d351a0459: Status 404 returned error can't find the container with id e056739b11dd57f4315f6486b888e0fa128eac9d43167e7563f1333d351a0459
Sep 08 13:23:42.741485 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:42.741444    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hp86f" event={"ID":"1b6be9ca-4f36-4dba-b1c0-9bf4487d2280","Type":"ContainerStarted","Data":"e056739b11dd57f4315f6486b888e0fa128eac9d43167e7563f1333d351a0459"}
Sep 08 13:23:44.751331 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:44.751275    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hp86f" event={"ID":"1b6be9ca-4f36-4dba-b1c0-9bf4487d2280","Type":"ContainerStarted","Data":"b1662f72d3d2eefab2375efec0ee40b7199df4a45b883dc773a740f0316570bc"}
Sep 08 13:23:44.775114 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:44.775043    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hp86f" podStartSLOduration=2.179924162 podStartE2EDuration="3.775021747s" totalImagesPullingTime="1.595097585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:23:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:23:44.774683137 +0000 UTC m=+456.235327947" watchObservedRunningTime="2026-09-08 13:23:44.775021747 +0000 UTC m=+456.235666557"
Sep 08 13:23:45.294330 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:45.294271    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rbfhv"]
Sep 08 13:23:45.296423 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:45.296396    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-rbfhv"
Sep 08 13:23:45.299010 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:45.298989    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 08 13:23:45.299271 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:45.299253    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6x58g\""
Sep 08 13:23:45.299367 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:45.299350    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 08 13:23:45.310344 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:45.310319    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rbfhv"]
Sep 08 13:23:45.419579 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:45.419541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a378acaf-2dab-4fa3-be38-16ac22ac2c59-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rbfhv\" (UID: \"a378acaf-2dab-4fa3-be38-16ac22ac2c59\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rbfhv"
Sep 08 13:23:45.419730 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:45.419600    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62mp7\" (UniqueName: \"kubernetes.io/projected/a378acaf-2dab-4fa3-be38-16ac22ac2c59-kube-api-access-62mp7\") pod \"cert-manager-webhook-6486ffd997-rbfhv\" (UID: \"a378acaf-2dab-4fa3-be38-16ac22ac2c59\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rbfhv"
Sep 08 13:23:45.520881 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:45.520842    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a378acaf-2dab-4fa3-be38-16ac22ac2c59-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rbfhv\" (UID: \"a378acaf-2dab-4fa3-be38-16ac22ac2c59\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rbfhv"
Sep 08 13:23:45.521083 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:45.520898    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62mp7\" (UniqueName: \"kubernetes.io/projected/a378acaf-2dab-4fa3-be38-16ac22ac2c59-kube-api-access-62mp7\") pod \"cert-manager-webhook-6486ffd997-rbfhv\" (UID: \"a378acaf-2dab-4fa3-be38-16ac22ac2c59\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rbfhv"
Sep 08 13:23:45.530223 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:45.530194    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62mp7\" (UniqueName: \"kubernetes.io/projected/a378acaf-2dab-4fa3-be38-16ac22ac2c59-kube-api-access-62mp7\") pod \"cert-manager-webhook-6486ffd997-rbfhv\" (UID: \"a378acaf-2dab-4fa3-be38-16ac22ac2c59\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rbfhv"
Sep 08 13:23:45.530488 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:45.530468    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a378acaf-2dab-4fa3-be38-16ac22ac2c59-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rbfhv\" (UID: \"a378acaf-2dab-4fa3-be38-16ac22ac2c59\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rbfhv"
Sep 08 13:23:45.605732 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:45.605649    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-rbfhv"
Sep 08 13:23:45.884459 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:45.884377    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rbfhv"]
Sep 08 13:23:45.887243 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:23:45.887199    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda378acaf_2dab_4fa3_be38_16ac22ac2c59.slice/crio-7c8825ab64436cf263049382cf4ac4a0dbb69d842446a8ce3468b7b24d6cd8f3 WatchSource:0}: Error finding container 7c8825ab64436cf263049382cf4ac4a0dbb69d842446a8ce3468b7b24d6cd8f3: Status 404 returned error can't find the container with id 7c8825ab64436cf263049382cf4ac4a0dbb69d842446a8ce3468b7b24d6cd8f3
Sep 08 13:23:46.763984 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:46.763938    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-rbfhv" event={"ID":"a378acaf-2dab-4fa3-be38-16ac22ac2c59","Type":"ContainerStarted","Data":"7c8825ab64436cf263049382cf4ac4a0dbb69d842446a8ce3468b7b24d6cd8f3"}
Sep 08 13:23:48.775768 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:48.775737    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-rbfhv" event={"ID":"a378acaf-2dab-4fa3-be38-16ac22ac2c59","Type":"ContainerStarted","Data":"f19d88dfb1521ba51032ed94d0b5f42bb99a52740f9b5d5aa1b2d3dac53213b7"}
Sep 08 13:23:48.776166 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:48.775793    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-rbfhv"
Sep 08 13:23:48.809102 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:48.809043    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-rbfhv" podStartSLOduration=1.295185498 podStartE2EDuration="3.809026118s" totalImagesPullingTime="2.51384062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:23:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:23:48.807502412 +0000 UTC m=+460.268147225" watchObservedRunningTime="2026-09-08 13:23:48.809026118 +0000 UTC m=+460.269670928"
Sep 08 13:23:50.873039 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:50.873001    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f"]
Sep 08 13:23:50.875403 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:50.875383    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f"
Sep 08 13:23:50.877751 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:50.877727    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 08 13:23:50.878558 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:50.878536    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 08 13:23:50.878675 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:50.878635    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lwbc7\""
Sep 08 13:23:50.885353 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:50.885325    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f"]
Sep 08 13:23:50.968972 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:50.968932    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8257e547-ea1a-43b7-b78b-4efcf753b40c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f\" (UID: \"8257e547-ea1a-43b7-b78b-4efcf753b40c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f"
Sep 08 13:23:50.969144 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:50.969009    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8257e547-ea1a-43b7-b78b-4efcf753b40c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f\" (UID: \"8257e547-ea1a-43b7-b78b-4efcf753b40c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f"
Sep 08 13:23:50.969144 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:50.969045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfgvz\" (UniqueName: \"kubernetes.io/projected/8257e547-ea1a-43b7-b78b-4efcf753b40c-kube-api-access-hfgvz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f\" (UID: \"8257e547-ea1a-43b7-b78b-4efcf753b40c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f"
Sep 08 13:23:51.058628 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:51.058588    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:51.058800 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:51.058697    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:51.063458 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:51.063437    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:51.069994 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:51.069975    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8257e547-ea1a-43b7-b78b-4efcf753b40c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f\" (UID: \"8257e547-ea1a-43b7-b78b-4efcf753b40c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f"
Sep 08 13:23:51.070086 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:51.070037    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8257e547-ea1a-43b7-b78b-4efcf753b40c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f\" (UID: \"8257e547-ea1a-43b7-b78b-4efcf753b40c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f"
Sep 08 13:23:51.070086 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:51.070068    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfgvz\" (UniqueName: \"kubernetes.io/projected/8257e547-ea1a-43b7-b78b-4efcf753b40c-kube-api-access-hfgvz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f\" (UID: \"8257e547-ea1a-43b7-b78b-4efcf753b40c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f"
Sep 08 13:23:51.070425 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:51.070403    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8257e547-ea1a-43b7-b78b-4efcf753b40c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f\" (UID: \"8257e547-ea1a-43b7-b78b-4efcf753b40c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f"
Sep 08 13:23:51.070497 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:51.070437    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8257e547-ea1a-43b7-b78b-4efcf753b40c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f\" (UID: \"8257e547-ea1a-43b7-b78b-4efcf753b40c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f"
Sep 08 13:23:51.081915 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:51.081882    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfgvz\" (UniqueName: \"kubernetes.io/projected/8257e547-ea1a-43b7-b78b-4efcf753b40c-kube-api-access-hfgvz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f\" (UID: \"8257e547-ea1a-43b7-b78b-4efcf753b40c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f"
Sep 08 13:23:51.184607 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:51.184580    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f"
Sep 08 13:23:51.476847 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:51.476790    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f"]
Sep 08 13:23:51.479332 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:23:51.479304    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8257e547_ea1a_43b7_b78b_4efcf753b40c.slice/crio-53dd2cb38c902156d2fa9bcfc123624ed1fcabbc7082989171558fba8c01ea73 WatchSource:0}: Error finding container 53dd2cb38c902156d2fa9bcfc123624ed1fcabbc7082989171558fba8c01ea73: Status 404 returned error can't find the container with id 53dd2cb38c902156d2fa9bcfc123624ed1fcabbc7082989171558fba8c01ea73
Sep 08 13:23:51.789839 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:51.789745    2561 generic.go:356] "Generic (PLEG): container finished" podID="8257e547-ea1a-43b7-b78b-4efcf753b40c" containerID="87e5f7dbb866e581a9cf3e58654c1342bb8b554943c52aebaab04338187acd76" exitCode=0
Sep 08 13:23:51.789839 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:51.789826    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f" event={"ID":"8257e547-ea1a-43b7-b78b-4efcf753b40c","Type":"ContainerDied","Data":"87e5f7dbb866e581a9cf3e58654c1342bb8b554943c52aebaab04338187acd76"}
Sep 08 13:23:51.790044 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:51.789875    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f" event={"ID":"8257e547-ea1a-43b7-b78b-4efcf753b40c","Type":"ContainerStarted","Data":"53dd2cb38c902156d2fa9bcfc123624ed1fcabbc7082989171558fba8c01ea73"}
Sep 08 13:23:51.794236 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:51.794215    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-577596d6f-fsg59"
Sep 08 13:23:52.881400 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:52.881356    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wzhls"]
Sep 08 13:23:52.883759 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:52.883737    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wzhls"
Sep 08 13:23:52.886489 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:52.886464    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-f6b94\""
Sep 08 13:23:52.906089 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:52.906060    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wzhls"]
Sep 08 13:23:52.987406 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:52.987367    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d20f3e09-0fd8-4a80-9326-39248439bb2d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wzhls\" (UID: \"d20f3e09-0fd8-4a80-9326-39248439bb2d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wzhls"
Sep 08 13:23:52.987730 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:52.987701    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqcfq\" (UniqueName: \"kubernetes.io/projected/d20f3e09-0fd8-4a80-9326-39248439bb2d-kube-api-access-vqcfq\") pod \"cert-manager-cainjector-559cdd8c87-wzhls\" (UID: \"d20f3e09-0fd8-4a80-9326-39248439bb2d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wzhls"
Sep 08 13:23:53.089079 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:53.089044    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d20f3e09-0fd8-4a80-9326-39248439bb2d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wzhls\" (UID: \"d20f3e09-0fd8-4a80-9326-39248439bb2d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wzhls"
Sep 08 13:23:53.089257 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:53.089145    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqcfq\" (UniqueName: \"kubernetes.io/projected/d20f3e09-0fd8-4a80-9326-39248439bb2d-kube-api-access-vqcfq\") pod \"cert-manager-cainjector-559cdd8c87-wzhls\" (UID: \"d20f3e09-0fd8-4a80-9326-39248439bb2d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wzhls"
Sep 08 13:23:53.098172 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:53.098142    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d20f3e09-0fd8-4a80-9326-39248439bb2d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wzhls\" (UID: \"d20f3e09-0fd8-4a80-9326-39248439bb2d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wzhls"
Sep 08 13:23:53.098336 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:53.098216    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqcfq\" (UniqueName: \"kubernetes.io/projected/d20f3e09-0fd8-4a80-9326-39248439bb2d-kube-api-access-vqcfq\") pod \"cert-manager-cainjector-559cdd8c87-wzhls\" (UID: \"d20f3e09-0fd8-4a80-9326-39248439bb2d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wzhls"
Sep 08 13:23:53.192253 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:53.192220    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wzhls"
Sep 08 13:23:53.466318 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:53.466273    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wzhls"]
Sep 08 13:23:53.469091 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:23:53.469061    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd20f3e09_0fd8_4a80_9326_39248439bb2d.slice/crio-0e008d273eb7ebd62dce2a997c714acb101fe44a7b4404b4322fdf0f26078e1c WatchSource:0}: Error finding container 0e008d273eb7ebd62dce2a997c714acb101fe44a7b4404b4322fdf0f26078e1c: Status 404 returned error can't find the container with id 0e008d273eb7ebd62dce2a997c714acb101fe44a7b4404b4322fdf0f26078e1c
Sep 08 13:23:53.801231 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:53.801122    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wzhls" event={"ID":"d20f3e09-0fd8-4a80-9326-39248439bb2d","Type":"ContainerStarted","Data":"80810a3d79be19d4327ce04ed3e68f818c98cc09ca051108cb05f58c7e007677"}
Sep 08 13:23:53.801231 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:53.801176    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wzhls" event={"ID":"d20f3e09-0fd8-4a80-9326-39248439bb2d","Type":"ContainerStarted","Data":"0e008d273eb7ebd62dce2a997c714acb101fe44a7b4404b4322fdf0f26078e1c"}
Sep 08 13:23:53.828542 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:53.828482    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wzhls" podStartSLOduration=1.8284658299999998 podStartE2EDuration="1.82846583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:23:53.8270738 +0000 UTC m=+465.287718611" watchObservedRunningTime="2026-09-08 13:23:53.82846583 +0000 UTC m=+465.289110640"
Sep 08 13:23:54.782739 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:23:54.782703    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-rbfhv"
Sep 08 13:24:09.875065 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:09.875031    2561 generic.go:356] "Generic (PLEG): container finished" podID="8257e547-ea1a-43b7-b78b-4efcf753b40c" containerID="c66c2f566b559c05fc1ba59b35677dc6b44bfc0cc87598c2ec8322b7e6eff42b" exitCode=0
Sep 08 13:24:09.875479 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:09.875108    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f" event={"ID":"8257e547-ea1a-43b7-b78b-4efcf753b40c","Type":"ContainerDied","Data":"c66c2f566b559c05fc1ba59b35677dc6b44bfc0cc87598c2ec8322b7e6eff42b"}
Sep 08 13:24:10.881715 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:10.881683    2561 generic.go:356] "Generic (PLEG): container finished" podID="8257e547-ea1a-43b7-b78b-4efcf753b40c" containerID="9e98d6fc51617d24f10b3123d76976bd8bbd34d20f5cd87d2d0b6e8ddc6c79d1" exitCode=0
Sep 08 13:24:10.882098 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:10.881769    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f" event={"ID":"8257e547-ea1a-43b7-b78b-4efcf753b40c","Type":"ContainerDied","Data":"9e98d6fc51617d24f10b3123d76976bd8bbd34d20f5cd87d2d0b6e8ddc6c79d1"}
Sep 08 13:24:12.009954 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:12.009927    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f"
Sep 08 13:24:12.043567 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:12.043539    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8257e547-ea1a-43b7-b78b-4efcf753b40c-util\") pod \"8257e547-ea1a-43b7-b78b-4efcf753b40c\" (UID: \"8257e547-ea1a-43b7-b78b-4efcf753b40c\") "
Sep 08 13:24:12.043729 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:12.043595    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8257e547-ea1a-43b7-b78b-4efcf753b40c-bundle\") pod \"8257e547-ea1a-43b7-b78b-4efcf753b40c\" (UID: \"8257e547-ea1a-43b7-b78b-4efcf753b40c\") "
Sep 08 13:24:12.043729 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:12.043638    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfgvz\" (UniqueName: \"kubernetes.io/projected/8257e547-ea1a-43b7-b78b-4efcf753b40c-kube-api-access-hfgvz\") pod \"8257e547-ea1a-43b7-b78b-4efcf753b40c\" (UID: \"8257e547-ea1a-43b7-b78b-4efcf753b40c\") "
Sep 08 13:24:12.044030 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:12.044003    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8257e547-ea1a-43b7-b78b-4efcf753b40c-bundle" (OuterVolumeSpecName: "bundle") pod "8257e547-ea1a-43b7-b78b-4efcf753b40c" (UID: "8257e547-ea1a-43b7-b78b-4efcf753b40c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:24:12.045750 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:12.045719    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8257e547-ea1a-43b7-b78b-4efcf753b40c-kube-api-access-hfgvz" (OuterVolumeSpecName: "kube-api-access-hfgvz") pod "8257e547-ea1a-43b7-b78b-4efcf753b40c" (UID: "8257e547-ea1a-43b7-b78b-4efcf753b40c"). InnerVolumeSpecName "kube-api-access-hfgvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:24:12.048688 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:12.048663    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8257e547-ea1a-43b7-b78b-4efcf753b40c-util" (OuterVolumeSpecName: "util") pod "8257e547-ea1a-43b7-b78b-4efcf753b40c" (UID: "8257e547-ea1a-43b7-b78b-4efcf753b40c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:24:12.145051 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:12.144961    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8257e547-ea1a-43b7-b78b-4efcf753b40c-bundle\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:24:12.145051 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:12.144990    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfgvz\" (UniqueName: \"kubernetes.io/projected/8257e547-ea1a-43b7-b78b-4efcf753b40c-kube-api-access-hfgvz\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:24:12.145051 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:12.145003    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8257e547-ea1a-43b7-b78b-4efcf753b40c-util\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:24:12.892500 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:12.892472    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f"
Sep 08 13:24:12.892500 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:12.892481    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkfb6f" event={"ID":"8257e547-ea1a-43b7-b78b-4efcf753b40c","Type":"ContainerDied","Data":"53dd2cb38c902156d2fa9bcfc123624ed1fcabbc7082989171558fba8c01ea73"}
Sep 08 13:24:12.892699 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:12.892521    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53dd2cb38c902156d2fa9bcfc123624ed1fcabbc7082989171558fba8c01ea73"
Sep 08 13:24:30.585328 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.585282    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"]
Sep 08 13:24:30.585781 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.585585    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8257e547-ea1a-43b7-b78b-4efcf753b40c" containerName="util"
Sep 08 13:24:30.585781 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.585595    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8257e547-ea1a-43b7-b78b-4efcf753b40c" containerName="util"
Sep 08 13:24:30.585781 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.585607    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8257e547-ea1a-43b7-b78b-4efcf753b40c" containerName="extract"
Sep 08 13:24:30.585781 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.585613    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8257e547-ea1a-43b7-b78b-4efcf753b40c" containerName="extract"
Sep 08 13:24:30.585781 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.585624    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8257e547-ea1a-43b7-b78b-4efcf753b40c" containerName="pull"
Sep 08 13:24:30.585781 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.585629    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8257e547-ea1a-43b7-b78b-4efcf753b40c" containerName="pull"
Sep 08 13:24:30.585781 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.585682    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8257e547-ea1a-43b7-b78b-4efcf753b40c" containerName="extract"
Sep 08 13:24:30.589625 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.589607    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.593956 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.593935    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 08 13:24:30.594069 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.593936    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 08 13:24:30.594115 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.594067    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 08 13:24:30.594682 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.594666    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bjkjl\""
Sep 08 13:24:30.594912 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.594895    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 13:24:30.595574 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.595550    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 08 13:24:30.595684 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.595634    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 13:24:30.611780 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.611754    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"]
Sep 08 13:24:30.697106 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.697068    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be5770ee-8459-4a0b-840e-a93aeaf3603a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-s5gnb\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.697253 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.697143    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-s5gnb\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.697253 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.697179    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd6vj\" (UniqueName: \"kubernetes.io/projected/be5770ee-8459-4a0b-840e-a93aeaf3603a-kube-api-access-hd6vj\") pod \"istiod-openshift-gateway-7b7995d5cd-s5gnb\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.697253 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.697197    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-s5gnb\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.697253 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.697215    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-s5gnb\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.697456 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.697340    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5770ee-8459-4a0b-840e-a93aeaf3603a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-s5gnb\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.697456 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.697372    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-s5gnb\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.798182 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.798138    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be5770ee-8459-4a0b-840e-a93aeaf3603a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-s5gnb\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.798417 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.798224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-s5gnb\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.798417 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.798263    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd6vj\" (UniqueName: \"kubernetes.io/projected/be5770ee-8459-4a0b-840e-a93aeaf3603a-kube-api-access-hd6vj\") pod \"istiod-openshift-gateway-7b7995d5cd-s5gnb\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.798417 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.798324    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-s5gnb\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.798417 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.798353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-s5gnb\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.798417 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.798388    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5770ee-8459-4a0b-840e-a93aeaf3603a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-s5gnb\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.798417 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.798411    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-s5gnb\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.798941 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.798908    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-s5gnb\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.800755 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.800712    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5770ee-8459-4a0b-840e-a93aeaf3603a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-s5gnb\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.800872 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.800779    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-s5gnb\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.800981 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.800963    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-s5gnb\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.801084 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.801067    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be5770ee-8459-4a0b-840e-a93aeaf3603a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-s5gnb\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.831946 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.831912    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-s5gnb\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.846650 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.846580    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd6vj\" (UniqueName: \"kubernetes.io/projected/be5770ee-8459-4a0b-840e-a93aeaf3603a-kube-api-access-hd6vj\") pod \"istiod-openshift-gateway-7b7995d5cd-s5gnb\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:30.899212 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:30.899171    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:31.192082 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:31.192051    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"]
Sep 08 13:24:31.194955 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:24:31.194928    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5770ee_8459_4a0b_840e_a93aeaf3603a.slice/crio-3d12ffeaa0f949465a17c96fffedd7648b9c55b1d3dc1c312af339bab46d3437 WatchSource:0}: Error finding container 3d12ffeaa0f949465a17c96fffedd7648b9c55b1d3dc1c312af339bab46d3437: Status 404 returned error can't find the container with id 3d12ffeaa0f949465a17c96fffedd7648b9c55b1d3dc1c312af339bab46d3437
Sep 08 13:24:31.989731 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:31.989687    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb" event={"ID":"be5770ee-8459-4a0b-840e-a93aeaf3603a","Type":"ContainerStarted","Data":"3d12ffeaa0f949465a17c96fffedd7648b9c55b1d3dc1c312af339bab46d3437"}
Sep 08 13:24:34.116268 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:34.116172    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 13:24:34.116268 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:34.116243    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 13:24:35.007529 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:35.007494    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb" event={"ID":"be5770ee-8459-4a0b-840e-a93aeaf3603a","Type":"ContainerStarted","Data":"d3d46048481af486a41ef8c67ca2dd04d4636333776d66705a0ee9b0a552f839"}
Sep 08 13:24:35.007737 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:35.007671    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:35.009478 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:35.009452    2561 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-s5gnb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 08 13:24:35.009571 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:35.009501    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb" podUID="be5770ee-8459-4a0b-840e-a93aeaf3603a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:24:35.045841 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:35.045788    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb" podStartSLOduration=2.126847217 podStartE2EDuration="5.045770612s" totalImagesPullingTime="2.918923395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:24:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:24:35.044353671 +0000 UTC m=+506.504998472" watchObservedRunningTime="2026-09-08 13:24:35.045770612 +0000 UTC m=+506.506415422"
Sep 08 13:24:36.012776 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:36.012746    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:24:37.448964 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.448916    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"]
Sep 08 13:24:37.461020 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.460983    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.463175 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.463150    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-944bd\""
Sep 08 13:24:37.469251 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.469228    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"]
Sep 08 13:24:37.559601 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.559562    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8dab86-bf9e-4760-8002-d3a068f8d45d-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.559791 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.559609    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8dab86-bf9e-4760-8002-d3a068f8d45d-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.559791 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.559633    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8dab86-bf9e-4760-8002-d3a068f8d45d-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.559791 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.559689    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8dab86-bf9e-4760-8002-d3a068f8d45d-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.559791 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.559704    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8dab86-bf9e-4760-8002-d3a068f8d45d-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.559791 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.559770    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8dab86-bf9e-4760-8002-d3a068f8d45d-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.560026 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.559873    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8dab86-bf9e-4760-8002-d3a068f8d45d-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.560026 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.559912    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8dab86-bf9e-4760-8002-d3a068f8d45d-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.560026 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.559939    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c5z4\" (UniqueName: \"kubernetes.io/projected/db8dab86-bf9e-4760-8002-d3a068f8d45d-kube-api-access-9c5z4\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.660626 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.660589    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8dab86-bf9e-4760-8002-d3a068f8d45d-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.660829 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.660638    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8dab86-bf9e-4760-8002-d3a068f8d45d-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.660829 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.660664    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8dab86-bf9e-4760-8002-d3a068f8d45d-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.660829 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.660702    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8dab86-bf9e-4760-8002-d3a068f8d45d-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.660829 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.660725    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8dab86-bf9e-4760-8002-d3a068f8d45d-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.660829 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.660765    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8dab86-bf9e-4760-8002-d3a068f8d45d-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.660829 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.660810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8dab86-bf9e-4760-8002-d3a068f8d45d-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.660829 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.660831    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8dab86-bf9e-4760-8002-d3a068f8d45d-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.661742 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.660853    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c5z4\" (UniqueName: \"kubernetes.io/projected/db8dab86-bf9e-4760-8002-d3a068f8d45d-kube-api-access-9c5z4\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.661742 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.661182    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8dab86-bf9e-4760-8002-d3a068f8d45d-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.661742 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.661263    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8dab86-bf9e-4760-8002-d3a068f8d45d-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.661742 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.661312    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8dab86-bf9e-4760-8002-d3a068f8d45d-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.661742 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.661484    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8dab86-bf9e-4760-8002-d3a068f8d45d-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.661742 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.661510    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8dab86-bf9e-4760-8002-d3a068f8d45d-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.663527 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.663503    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8dab86-bf9e-4760-8002-d3a068f8d45d-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.663763 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.663747    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8dab86-bf9e-4760-8002-d3a068f8d45d-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.678716 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.678691    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8dab86-bf9e-4760-8002-d3a068f8d45d-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.678862 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.678842    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c5z4\" (UniqueName: \"kubernetes.io/projected/db8dab86-bf9e-4760-8002-d3a068f8d45d-kube-api-access-9c5z4\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-b5cvn\" (UID: \"db8dab86-bf9e-4760-8002-d3a068f8d45d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:37.773743 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:37.773645    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:38.050181 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:38.050154    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"]
Sep 08 13:24:38.052156 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:24:38.052131    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8dab86_bf9e_4760_8002_d3a068f8d45d.slice/crio-18bd09b353b7538307c68d92edf96a9c824f3f77c36dc96e38a645c40cfdb21d WatchSource:0}: Error finding container 18bd09b353b7538307c68d92edf96a9c824f3f77c36dc96e38a645c40cfdb21d: Status 404 returned error can't find the container with id 18bd09b353b7538307c68d92edf96a9c824f3f77c36dc96e38a645c40cfdb21d
Sep 08 13:24:39.027924 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:39.027876    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn" event={"ID":"db8dab86-bf9e-4760-8002-d3a068f8d45d","Type":"ContainerStarted","Data":"18bd09b353b7538307c68d92edf96a9c824f3f77c36dc96e38a645c40cfdb21d"}
Sep 08 13:24:40.825948 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:40.825914    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 13:24:40.826217 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:40.825985    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 13:24:40.826217 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:40.826016    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 13:24:41.039647 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:41.039606    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn" event={"ID":"db8dab86-bf9e-4760-8002-d3a068f8d45d","Type":"ContainerStarted","Data":"0f0f9ce4ea1aba582a7f5c141712c08746097b6feb1e6b863062d06922781fef"}
Sep 08 13:24:41.061399 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:41.061327    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn" podStartSLOduration=1.289773782 podStartE2EDuration="4.061305313s" totalImagesPullingTime="2.771531531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:24:41.060743818 +0000 UTC m=+512.521388628" watchObservedRunningTime="2026-09-08 13:24:41.061305313 +0000 UTC m=+512.521950120"
Sep 08 13:24:41.774622 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:41.774587    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:41.779203 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:41.779174    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:42.043913 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:42.043822    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:42.044962 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:42.044937    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:24:46.920370 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:46.920276    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr"]
Sep 08 13:24:46.923756 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:46.923735    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr"
Sep 08 13:24:46.926468 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:46.926444    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lwbc7\""
Sep 08 13:24:46.927121 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:46.927101    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 08 13:24:46.927121 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:46.927111    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 08 13:24:46.934274 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:46.934249    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr"]
Sep 08 13:24:47.035373 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.035342    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv"]
Sep 08 13:24:47.038711 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.038695    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv"
Sep 08 13:24:47.047040 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.047016    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09d6a572-7d35-4acd-beb8-edc776fa9033-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr\" (UID: \"09d6a572-7d35-4acd-beb8-edc776fa9033\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr"
Sep 08 13:24:47.047308 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.047256    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx5hp\" (UniqueName: \"kubernetes.io/projected/09d6a572-7d35-4acd-beb8-edc776fa9033-kube-api-access-dx5hp\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr\" (UID: \"09d6a572-7d35-4acd-beb8-edc776fa9033\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr"
Sep 08 13:24:47.047417 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.047329    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09d6a572-7d35-4acd-beb8-edc776fa9033-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr\" (UID: \"09d6a572-7d35-4acd-beb8-edc776fa9033\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr"
Sep 08 13:24:47.049398 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.049375    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv"]
Sep 08 13:24:47.130557 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.130518    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt"]
Sep 08 13:24:47.133939 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.133921    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt"
Sep 08 13:24:47.147460 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.147426    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt"]
Sep 08 13:24:47.147737 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.147717    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09a98cce-8483-4e68-a01a-5b0f024cce5c-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv\" (UID: \"09a98cce-8483-4e68-a01a-5b0f024cce5c\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv"
Sep 08 13:24:47.147795 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.147748    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2zbx\" (UniqueName: \"kubernetes.io/projected/09a98cce-8483-4e68-a01a-5b0f024cce5c-kube-api-access-r2zbx\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv\" (UID: \"09a98cce-8483-4e68-a01a-5b0f024cce5c\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv"
Sep 08 13:24:47.147795 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.147775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09a98cce-8483-4e68-a01a-5b0f024cce5c-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv\" (UID: \"09a98cce-8483-4e68-a01a-5b0f024cce5c\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv"
Sep 08 13:24:47.147879 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.147833    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09d6a572-7d35-4acd-beb8-edc776fa9033-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr\" (UID: \"09d6a572-7d35-4acd-beb8-edc776fa9033\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr"
Sep 08 13:24:47.147879 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.147868    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx5hp\" (UniqueName: \"kubernetes.io/projected/09d6a572-7d35-4acd-beb8-edc776fa9033-kube-api-access-dx5hp\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr\" (UID: \"09d6a572-7d35-4acd-beb8-edc776fa9033\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr"
Sep 08 13:24:47.147976 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.147902    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09d6a572-7d35-4acd-beb8-edc776fa9033-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr\" (UID: \"09d6a572-7d35-4acd-beb8-edc776fa9033\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr"
Sep 08 13:24:47.148278 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.148249    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09d6a572-7d35-4acd-beb8-edc776fa9033-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr\" (UID: \"09d6a572-7d35-4acd-beb8-edc776fa9033\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr"
Sep 08 13:24:47.148278 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.148267    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09d6a572-7d35-4acd-beb8-edc776fa9033-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr\" (UID: \"09d6a572-7d35-4acd-beb8-edc776fa9033\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr"
Sep 08 13:24:47.157056 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.157019    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx5hp\" (UniqueName: \"kubernetes.io/projected/09d6a572-7d35-4acd-beb8-edc776fa9033-kube-api-access-dx5hp\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr\" (UID: \"09d6a572-7d35-4acd-beb8-edc776fa9033\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr"
Sep 08 13:24:47.230782 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.230750    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq"]
Sep 08 13:24:47.232821 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.232802    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr"
Sep 08 13:24:47.234321 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.234278    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq"
Sep 08 13:24:47.249034 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.249004    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/233af0ef-37ac-41c9-a338-3f8fbd4814db-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt\" (UID: \"233af0ef-37ac-41c9-a338-3f8fbd4814db\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt"
Sep 08 13:24:47.249169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.249048    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/233af0ef-37ac-41c9-a338-3f8fbd4814db-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt\" (UID: \"233af0ef-37ac-41c9-a338-3f8fbd4814db\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt"
Sep 08 13:24:47.249169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.249099    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p62gj\" (UniqueName: \"kubernetes.io/projected/233af0ef-37ac-41c9-a338-3f8fbd4814db-kube-api-access-p62gj\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt\" (UID: \"233af0ef-37ac-41c9-a338-3f8fbd4814db\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt"
Sep 08 13:24:47.249169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.249141    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09a98cce-8483-4e68-a01a-5b0f024cce5c-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv\" (UID: \"09a98cce-8483-4e68-a01a-5b0f024cce5c\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv"
Sep 08 13:24:47.249169 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.249164    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2zbx\" (UniqueName: \"kubernetes.io/projected/09a98cce-8483-4e68-a01a-5b0f024cce5c-kube-api-access-r2zbx\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv\" (UID: \"09a98cce-8483-4e68-a01a-5b0f024cce5c\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv"
Sep 08 13:24:47.249432 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.249190    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09a98cce-8483-4e68-a01a-5b0f024cce5c-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv\" (UID: \"09a98cce-8483-4e68-a01a-5b0f024cce5c\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv"
Sep 08 13:24:47.249432 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.249254    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq"]
Sep 08 13:24:47.250353 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.249624    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09a98cce-8483-4e68-a01a-5b0f024cce5c-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv\" (UID: \"09a98cce-8483-4e68-a01a-5b0f024cce5c\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv"
Sep 08 13:24:47.250353 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.249783    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09a98cce-8483-4e68-a01a-5b0f024cce5c-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv\" (UID: \"09a98cce-8483-4e68-a01a-5b0f024cce5c\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv"
Sep 08 13:24:47.258522 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.258496    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2zbx\" (UniqueName: \"kubernetes.io/projected/09a98cce-8483-4e68-a01a-5b0f024cce5c-kube-api-access-r2zbx\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv\" (UID: \"09a98cce-8483-4e68-a01a-5b0f024cce5c\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv"
Sep 08 13:24:47.347615 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.347576    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv"
Sep 08 13:24:47.349966 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.349937    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm74s\" (UniqueName: \"kubernetes.io/projected/5928638b-1cf2-4486-a94f-182853c798d3-kube-api-access-tm74s\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq\" (UID: \"5928638b-1cf2-4486-a94f-182853c798d3\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq"
Sep 08 13:24:47.350042 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.349988    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/233af0ef-37ac-41c9-a338-3f8fbd4814db-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt\" (UID: \"233af0ef-37ac-41c9-a338-3f8fbd4814db\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt"
Sep 08 13:24:47.350042 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.350011    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/233af0ef-37ac-41c9-a338-3f8fbd4814db-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt\" (UID: \"233af0ef-37ac-41c9-a338-3f8fbd4814db\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt"
Sep 08 13:24:47.350118 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.350069    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p62gj\" (UniqueName: \"kubernetes.io/projected/233af0ef-37ac-41c9-a338-3f8fbd4814db-kube-api-access-p62gj\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt\" (UID: \"233af0ef-37ac-41c9-a338-3f8fbd4814db\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt"
Sep 08 13:24:47.350162 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.350134    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5928638b-1cf2-4486-a94f-182853c798d3-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq\" (UID: \"5928638b-1cf2-4486-a94f-182853c798d3\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq"
Sep 08 13:24:47.350198 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.350166    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5928638b-1cf2-4486-a94f-182853c798d3-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq\" (UID: \"5928638b-1cf2-4486-a94f-182853c798d3\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq"
Sep 08 13:24:47.350358 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.350340    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/233af0ef-37ac-41c9-a338-3f8fbd4814db-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt\" (UID: \"233af0ef-37ac-41c9-a338-3f8fbd4814db\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt"
Sep 08 13:24:47.350408 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.350387    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/233af0ef-37ac-41c9-a338-3f8fbd4814db-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt\" (UID: \"233af0ef-37ac-41c9-a338-3f8fbd4814db\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt"
Sep 08 13:24:47.359482 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.359457    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p62gj\" (UniqueName: \"kubernetes.io/projected/233af0ef-37ac-41c9-a338-3f8fbd4814db-kube-api-access-p62gj\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt\" (UID: \"233af0ef-37ac-41c9-a338-3f8fbd4814db\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt"
Sep 08 13:24:47.442579 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.442547    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt"
Sep 08 13:24:47.450915 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.450890    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5928638b-1cf2-4486-a94f-182853c798d3-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq\" (UID: \"5928638b-1cf2-4486-a94f-182853c798d3\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq"
Sep 08 13:24:47.451035 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.450929    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5928638b-1cf2-4486-a94f-182853c798d3-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq\" (UID: \"5928638b-1cf2-4486-a94f-182853c798d3\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq"
Sep 08 13:24:47.451035 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.451008    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm74s\" (UniqueName: \"kubernetes.io/projected/5928638b-1cf2-4486-a94f-182853c798d3-kube-api-access-tm74s\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq\" (UID: \"5928638b-1cf2-4486-a94f-182853c798d3\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq"
Sep 08 13:24:47.457253 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.451550    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5928638b-1cf2-4486-a94f-182853c798d3-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq\" (UID: \"5928638b-1cf2-4486-a94f-182853c798d3\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq"
Sep 08 13:24:47.457253 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.451673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5928638b-1cf2-4486-a94f-182853c798d3-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq\" (UID: \"5928638b-1cf2-4486-a94f-182853c798d3\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq"
Sep 08 13:24:47.474156 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.474126    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm74s\" (UniqueName: \"kubernetes.io/projected/5928638b-1cf2-4486-a94f-182853c798d3-kube-api-access-tm74s\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq\" (UID: \"5928638b-1cf2-4486-a94f-182853c798d3\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq"
Sep 08 13:24:47.516523 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.516490    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr"]
Sep 08 13:24:47.519585 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:24:47.519550    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d6a572_7d35_4acd_beb8_edc776fa9033.slice/crio-8e7819f725e713eadeac4a3f8cccd37ad626d4e4e0179949ed9302898116eda8 WatchSource:0}: Error finding container 8e7819f725e713eadeac4a3f8cccd37ad626d4e4e0179949ed9302898116eda8: Status 404 returned error can't find the container with id 8e7819f725e713eadeac4a3f8cccd37ad626d4e4e0179949ed9302898116eda8
Sep 08 13:24:47.561099 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.561070    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq"
Sep 08 13:24:47.626563 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.626484    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv"]
Sep 08 13:24:47.676090 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:24:47.676056    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a98cce_8483_4e68_a01a_5b0f024cce5c.slice/crio-d799252d19e8e0d8cf10baa7ae1980fdbb10c4e9f707e808d919713228b0aeee WatchSource:0}: Error finding container d799252d19e8e0d8cf10baa7ae1980fdbb10c4e9f707e808d919713228b0aeee: Status 404 returned error can't find the container with id d799252d19e8e0d8cf10baa7ae1980fdbb10c4e9f707e808d919713228b0aeee
Sep 08 13:24:47.719341 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.719272    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt"]
Sep 08 13:24:47.721022 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:24:47.720998    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233af0ef_37ac_41c9_a338_3f8fbd4814db.slice/crio-8fefcdc5b5f683d39844625dad24cf217311ca214e2282363483586361e64f7e WatchSource:0}: Error finding container 8fefcdc5b5f683d39844625dad24cf217311ca214e2282363483586361e64f7e: Status 404 returned error can't find the container with id 8fefcdc5b5f683d39844625dad24cf217311ca214e2282363483586361e64f7e
Sep 08 13:24:47.858607 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:47.858579    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq"]
Sep 08 13:24:47.905170 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:24:47.905132    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5928638b_1cf2_4486_a94f_182853c798d3.slice/crio-052f76a3b191c6e7b267723180e0a74419accc1fc3a100453df53536aa630b30 WatchSource:0}: Error finding container 052f76a3b191c6e7b267723180e0a74419accc1fc3a100453df53536aa630b30: Status 404 returned error can't find the container with id 052f76a3b191c6e7b267723180e0a74419accc1fc3a100453df53536aa630b30
Sep 08 13:24:48.073631 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:48.073539    2561 generic.go:356] "Generic (PLEG): container finished" podID="09d6a572-7d35-4acd-beb8-edc776fa9033" containerID="5d46971e1823f60a37b6d92e0330880fc1367a85c69eaefc5f2827f7c33cb1bc" exitCode=0
Sep 08 13:24:48.073631 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:48.073611    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr" event={"ID":"09d6a572-7d35-4acd-beb8-edc776fa9033","Type":"ContainerDied","Data":"5d46971e1823f60a37b6d92e0330880fc1367a85c69eaefc5f2827f7c33cb1bc"}
Sep 08 13:24:48.074131 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:48.073646    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr" event={"ID":"09d6a572-7d35-4acd-beb8-edc776fa9033","Type":"ContainerStarted","Data":"8e7819f725e713eadeac4a3f8cccd37ad626d4e4e0179949ed9302898116eda8"}
Sep 08 13:24:48.075172 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:48.075148    2561 generic.go:356] "Generic (PLEG): container finished" podID="5928638b-1cf2-4486-a94f-182853c798d3" containerID="2505a1a19928a6e834d0b7d299ab9058c938e6859a3d067321a419b4ea320804" exitCode=0
Sep 08 13:24:48.075336 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:48.075215    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq" event={"ID":"5928638b-1cf2-4486-a94f-182853c798d3","Type":"ContainerDied","Data":"2505a1a19928a6e834d0b7d299ab9058c938e6859a3d067321a419b4ea320804"}
Sep 08 13:24:48.075336 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:48.075264    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq" event={"ID":"5928638b-1cf2-4486-a94f-182853c798d3","Type":"ContainerStarted","Data":"052f76a3b191c6e7b267723180e0a74419accc1fc3a100453df53536aa630b30"}
Sep 08 13:24:48.076884 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:48.076860    2561 generic.go:356] "Generic (PLEG): container finished" podID="233af0ef-37ac-41c9-a338-3f8fbd4814db" containerID="d3c6e8481f622fbcd187ab1ba89f42ed4fb81bfa8451ec98f9adeb34cbdb1338" exitCode=0
Sep 08 13:24:48.076995 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:48.076940    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt" event={"ID":"233af0ef-37ac-41c9-a338-3f8fbd4814db","Type":"ContainerDied","Data":"d3c6e8481f622fbcd187ab1ba89f42ed4fb81bfa8451ec98f9adeb34cbdb1338"}
Sep 08 13:24:48.076995 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:48.076985    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt" event={"ID":"233af0ef-37ac-41c9-a338-3f8fbd4814db","Type":"ContainerStarted","Data":"8fefcdc5b5f683d39844625dad24cf217311ca214e2282363483586361e64f7e"}
Sep 08 13:24:48.078830 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:48.078812    2561 generic.go:356] "Generic (PLEG): container finished" podID="09a98cce-8483-4e68-a01a-5b0f024cce5c" containerID="5f3c92a44478dd860618d3976c6684981928483b70bb80bbb2f6e1da10f85436" exitCode=0
Sep 08 13:24:48.078923 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:48.078862    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv" event={"ID":"09a98cce-8483-4e68-a01a-5b0f024cce5c","Type":"ContainerDied","Data":"5f3c92a44478dd860618d3976c6684981928483b70bb80bbb2f6e1da10f85436"}
Sep 08 13:24:48.078923 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:48.078904    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv" event={"ID":"09a98cce-8483-4e68-a01a-5b0f024cce5c","Type":"ContainerStarted","Data":"d799252d19e8e0d8cf10baa7ae1980fdbb10c4e9f707e808d919713228b0aeee"}
Sep 08 13:24:49.085363 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:49.085331    2561 generic.go:356] "Generic (PLEG): container finished" podID="233af0ef-37ac-41c9-a338-3f8fbd4814db" containerID="846453f430901a7b804596486ee36451dfefc5f042e52a7a0fd84afafd37b95f" exitCode=0
Sep 08 13:24:49.085775 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:49.085370    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt" event={"ID":"233af0ef-37ac-41c9-a338-3f8fbd4814db","Type":"ContainerDied","Data":"846453f430901a7b804596486ee36451dfefc5f042e52a7a0fd84afafd37b95f"}
Sep 08 13:24:50.093545 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:50.093511    2561 generic.go:356] "Generic (PLEG): container finished" podID="233af0ef-37ac-41c9-a338-3f8fbd4814db" containerID="6620d0e646630930a1ed3e30fd3bbd156bb413a2d6dc24b145ff237b48a5ebf2" exitCode=0
Sep 08 13:24:50.093966 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:50.093591    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt" event={"ID":"233af0ef-37ac-41c9-a338-3f8fbd4814db","Type":"ContainerDied","Data":"6620d0e646630930a1ed3e30fd3bbd156bb413a2d6dc24b145ff237b48a5ebf2"}
Sep 08 13:24:50.095542 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:50.095516    2561 generic.go:356] "Generic (PLEG): container finished" podID="5928638b-1cf2-4486-a94f-182853c798d3" containerID="b1d08bcdafec229da4a5acf224e86ff36846a350873e081ee7701b9f84525c87" exitCode=0
Sep 08 13:24:50.095653 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:50.095552    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq" event={"ID":"5928638b-1cf2-4486-a94f-182853c798d3","Type":"ContainerDied","Data":"b1d08bcdafec229da4a5acf224e86ff36846a350873e081ee7701b9f84525c87"}
Sep 08 13:24:50.275962 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:24:50.275932    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5928638b_1cf2_4486_a94f_182853c798d3.slice/crio-conmon-6fa9c964c921ab67531b66263aa312698fa5dc83b47f0f79dcac3be55c84af29.scope\": RecentStats: unable to find data in memory cache]"
Sep 08 13:24:51.102535 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:51.102497    2561 generic.go:356] "Generic (PLEG): container finished" podID="5928638b-1cf2-4486-a94f-182853c798d3" containerID="6fa9c964c921ab67531b66263aa312698fa5dc83b47f0f79dcac3be55c84af29" exitCode=0
Sep 08 13:24:51.102967 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:51.102579    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq" event={"ID":"5928638b-1cf2-4486-a94f-182853c798d3","Type":"ContainerDied","Data":"6fa9c964c921ab67531b66263aa312698fa5dc83b47f0f79dcac3be55c84af29"}
Sep 08 13:24:51.235210 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:51.235183    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt"
Sep 08 13:24:51.284077 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:51.284048    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/233af0ef-37ac-41c9-a338-3f8fbd4814db-util\") pod \"233af0ef-37ac-41c9-a338-3f8fbd4814db\" (UID: \"233af0ef-37ac-41c9-a338-3f8fbd4814db\") "
Sep 08 13:24:51.284234 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:51.284121    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/233af0ef-37ac-41c9-a338-3f8fbd4814db-bundle\") pod \"233af0ef-37ac-41c9-a338-3f8fbd4814db\" (UID: \"233af0ef-37ac-41c9-a338-3f8fbd4814db\") "
Sep 08 13:24:51.284234 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:51.284177    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p62gj\" (UniqueName: \"kubernetes.io/projected/233af0ef-37ac-41c9-a338-3f8fbd4814db-kube-api-access-p62gj\") pod \"233af0ef-37ac-41c9-a338-3f8fbd4814db\" (UID: \"233af0ef-37ac-41c9-a338-3f8fbd4814db\") "
Sep 08 13:24:51.284701 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:51.284661    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233af0ef-37ac-41c9-a338-3f8fbd4814db-bundle" (OuterVolumeSpecName: "bundle") pod "233af0ef-37ac-41c9-a338-3f8fbd4814db" (UID: "233af0ef-37ac-41c9-a338-3f8fbd4814db"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:24:51.286259 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:51.286231    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233af0ef-37ac-41c9-a338-3f8fbd4814db-kube-api-access-p62gj" (OuterVolumeSpecName: "kube-api-access-p62gj") pod "233af0ef-37ac-41c9-a338-3f8fbd4814db" (UID: "233af0ef-37ac-41c9-a338-3f8fbd4814db"). InnerVolumeSpecName "kube-api-access-p62gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:24:51.289207 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:51.289172    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233af0ef-37ac-41c9-a338-3f8fbd4814db-util" (OuterVolumeSpecName: "util") pod "233af0ef-37ac-41c9-a338-3f8fbd4814db" (UID: "233af0ef-37ac-41c9-a338-3f8fbd4814db"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:24:51.384955 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:51.384862    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/233af0ef-37ac-41c9-a338-3f8fbd4814db-util\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:24:51.384955 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:51.384894    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/233af0ef-37ac-41c9-a338-3f8fbd4814db-bundle\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:24:51.384955 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:51.384904    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p62gj\" (UniqueName: \"kubernetes.io/projected/233af0ef-37ac-41c9-a338-3f8fbd4814db-kube-api-access-p62gj\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:24:52.109111 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:52.109079    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt"
Sep 08 13:24:52.109111 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:52.109093    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs9wmt" event={"ID":"233af0ef-37ac-41c9-a338-3f8fbd4814db","Type":"ContainerDied","Data":"8fefcdc5b5f683d39844625dad24cf217311ca214e2282363483586361e64f7e"}
Sep 08 13:24:52.109532 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:52.109143    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fefcdc5b5f683d39844625dad24cf217311ca214e2282363483586361e64f7e"
Sep 08 13:24:52.237202 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:52.237178    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq"
Sep 08 13:24:52.291445 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:52.291406    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tm74s\" (UniqueName: \"kubernetes.io/projected/5928638b-1cf2-4486-a94f-182853c798d3-kube-api-access-tm74s\") pod \"5928638b-1cf2-4486-a94f-182853c798d3\" (UID: \"5928638b-1cf2-4486-a94f-182853c798d3\") "
Sep 08 13:24:52.291638 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:52.291458    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5928638b-1cf2-4486-a94f-182853c798d3-bundle\") pod \"5928638b-1cf2-4486-a94f-182853c798d3\" (UID: \"5928638b-1cf2-4486-a94f-182853c798d3\") "
Sep 08 13:24:52.291638 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:52.291499    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5928638b-1cf2-4486-a94f-182853c798d3-util\") pod \"5928638b-1cf2-4486-a94f-182853c798d3\" (UID: \"5928638b-1cf2-4486-a94f-182853c798d3\") "
Sep 08 13:24:52.292174 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:52.292145    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5928638b-1cf2-4486-a94f-182853c798d3-bundle" (OuterVolumeSpecName: "bundle") pod "5928638b-1cf2-4486-a94f-182853c798d3" (UID: "5928638b-1cf2-4486-a94f-182853c798d3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:24:52.293726 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:52.293700    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5928638b-1cf2-4486-a94f-182853c798d3-kube-api-access-tm74s" (OuterVolumeSpecName: "kube-api-access-tm74s") pod "5928638b-1cf2-4486-a94f-182853c798d3" (UID: "5928638b-1cf2-4486-a94f-182853c798d3"). InnerVolumeSpecName "kube-api-access-tm74s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:24:52.296969 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:52.296943    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5928638b-1cf2-4486-a94f-182853c798d3-util" (OuterVolumeSpecName: "util") pod "5928638b-1cf2-4486-a94f-182853c798d3" (UID: "5928638b-1cf2-4486-a94f-182853c798d3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:24:52.392761 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:52.392665    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm74s\" (UniqueName: \"kubernetes.io/projected/5928638b-1cf2-4486-a94f-182853c798d3-kube-api-access-tm74s\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:24:52.392761 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:52.392697    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5928638b-1cf2-4486-a94f-182853c798d3-bundle\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:24:52.392761 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:52.392707    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5928638b-1cf2-4486-a94f-182853c798d3-util\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:24:53.116584 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:53.116536    2561 generic.go:356] "Generic (PLEG): container finished" podID="09d6a572-7d35-4acd-beb8-edc776fa9033" containerID="a974329004ab51925d5f1f0350185a2206fc60ccc3a73b082dd63ee2a389a1bc" exitCode=0
Sep 08 13:24:53.117042 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:53.116619    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr" event={"ID":"09d6a572-7d35-4acd-beb8-edc776fa9033","Type":"ContainerDied","Data":"a974329004ab51925d5f1f0350185a2206fc60ccc3a73b082dd63ee2a389a1bc"}
Sep 08 13:24:53.118648 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:53.118629    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq"
Sep 08 13:24:53.118741 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:53.118631    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dhhjq" event={"ID":"5928638b-1cf2-4486-a94f-182853c798d3","Type":"ContainerDied","Data":"052f76a3b191c6e7b267723180e0a74419accc1fc3a100453df53536aa630b30"}
Sep 08 13:24:53.118783 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:53.118742    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="052f76a3b191c6e7b267723180e0a74419accc1fc3a100453df53536aa630b30"
Sep 08 13:24:54.127149 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:54.127114    2561 generic.go:356] "Generic (PLEG): container finished" podID="09d6a572-7d35-4acd-beb8-edc776fa9033" containerID="b48f9f13b93da3e36b6abab870c0a16a2600ade60fb736a30ec8e4be770dfde9" exitCode=0
Sep 08 13:24:54.127577 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:54.127200    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr" event={"ID":"09d6a572-7d35-4acd-beb8-edc776fa9033","Type":"ContainerDied","Data":"b48f9f13b93da3e36b6abab870c0a16a2600ade60fb736a30ec8e4be770dfde9"}
Sep 08 13:24:55.271340 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:55.271313    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr"
Sep 08 13:24:55.317821 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:55.317789    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dx5hp\" (UniqueName: \"kubernetes.io/projected/09d6a572-7d35-4acd-beb8-edc776fa9033-kube-api-access-dx5hp\") pod \"09d6a572-7d35-4acd-beb8-edc776fa9033\" (UID: \"09d6a572-7d35-4acd-beb8-edc776fa9033\") "
Sep 08 13:24:55.317984 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:55.317877    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09d6a572-7d35-4acd-beb8-edc776fa9033-bundle\") pod \"09d6a572-7d35-4acd-beb8-edc776fa9033\" (UID: \"09d6a572-7d35-4acd-beb8-edc776fa9033\") "
Sep 08 13:24:55.317984 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:55.317904    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09d6a572-7d35-4acd-beb8-edc776fa9033-util\") pod \"09d6a572-7d35-4acd-beb8-edc776fa9033\" (UID: \"09d6a572-7d35-4acd-beb8-edc776fa9033\") "
Sep 08 13:24:55.318443 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:55.318415    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d6a572-7d35-4acd-beb8-edc776fa9033-bundle" (OuterVolumeSpecName: "bundle") pod "09d6a572-7d35-4acd-beb8-edc776fa9033" (UID: "09d6a572-7d35-4acd-beb8-edc776fa9033"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:24:55.319930 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:55.319903    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d6a572-7d35-4acd-beb8-edc776fa9033-kube-api-access-dx5hp" (OuterVolumeSpecName: "kube-api-access-dx5hp") pod "09d6a572-7d35-4acd-beb8-edc776fa9033" (UID: "09d6a572-7d35-4acd-beb8-edc776fa9033"). InnerVolumeSpecName "kube-api-access-dx5hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:24:55.322630 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:55.322584    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d6a572-7d35-4acd-beb8-edc776fa9033-util" (OuterVolumeSpecName: "util") pod "09d6a572-7d35-4acd-beb8-edc776fa9033" (UID: "09d6a572-7d35-4acd-beb8-edc776fa9033"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:24:55.418916 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:55.418826    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09d6a572-7d35-4acd-beb8-edc776fa9033-bundle\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:24:55.418916 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:55.418855    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09d6a572-7d35-4acd-beb8-edc776fa9033-util\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:24:55.418916 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:55.418864    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx5hp\" (UniqueName: \"kubernetes.io/projected/09d6a572-7d35-4acd-beb8-edc776fa9033-kube-api-access-dx5hp\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:24:56.142207 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:56.142177    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr"
Sep 08 13:24:56.142442 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:56.142246    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbjnsgr" event={"ID":"09d6a572-7d35-4acd-beb8-edc776fa9033","Type":"ContainerDied","Data":"8e7819f725e713eadeac4a3f8cccd37ad626d4e4e0179949ed9302898116eda8"}
Sep 08 13:24:56.142442 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:24:56.142317    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e7819f725e713eadeac4a3f8cccd37ad626d4e4e0179949ed9302898116eda8"
Sep 08 13:25:00.165368 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:00.165336    2561 generic.go:356] "Generic (PLEG): container finished" podID="09a98cce-8483-4e68-a01a-5b0f024cce5c" containerID="69031122157f303e4d0500ae33896ba876e4701fc46bbc0d2029e1deddc57f2a" exitCode=0
Sep 08 13:25:00.165811 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:00.165395    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv" event={"ID":"09a98cce-8483-4e68-a01a-5b0f024cce5c","Type":"ContainerDied","Data":"69031122157f303e4d0500ae33896ba876e4701fc46bbc0d2029e1deddc57f2a"}
Sep 08 13:25:01.172190 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:01.172103    2561 generic.go:356] "Generic (PLEG): container finished" podID="09a98cce-8483-4e68-a01a-5b0f024cce5c" containerID="ece94f5aaa4e3386931453945ef740ac2a0c86418d8e94760111ff4d90aa4aac" exitCode=0
Sep 08 13:25:01.172574 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:01.172189    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv" event={"ID":"09a98cce-8483-4e68-a01a-5b0f024cce5c","Type":"ContainerDied","Data":"ece94f5aaa4e3386931453945ef740ac2a0c86418d8e94760111ff4d90aa4aac"}
Sep 08 13:25:02.303017 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:02.302987    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv"
Sep 08 13:25:02.375405 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:02.375373    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09a98cce-8483-4e68-a01a-5b0f024cce5c-bundle\") pod \"09a98cce-8483-4e68-a01a-5b0f024cce5c\" (UID: \"09a98cce-8483-4e68-a01a-5b0f024cce5c\") "
Sep 08 13:25:02.375570 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:02.375427    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09a98cce-8483-4e68-a01a-5b0f024cce5c-util\") pod \"09a98cce-8483-4e68-a01a-5b0f024cce5c\" (UID: \"09a98cce-8483-4e68-a01a-5b0f024cce5c\") "
Sep 08 13:25:02.375570 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:02.375484    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2zbx\" (UniqueName: \"kubernetes.io/projected/09a98cce-8483-4e68-a01a-5b0f024cce5c-kube-api-access-r2zbx\") pod \"09a98cce-8483-4e68-a01a-5b0f024cce5c\" (UID: \"09a98cce-8483-4e68-a01a-5b0f024cce5c\") "
Sep 08 13:25:02.375924 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:02.375891    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a98cce-8483-4e68-a01a-5b0f024cce5c-bundle" (OuterVolumeSpecName: "bundle") pod "09a98cce-8483-4e68-a01a-5b0f024cce5c" (UID: "09a98cce-8483-4e68-a01a-5b0f024cce5c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:25:02.377737 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:02.377715    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a98cce-8483-4e68-a01a-5b0f024cce5c-kube-api-access-r2zbx" (OuterVolumeSpecName: "kube-api-access-r2zbx") pod "09a98cce-8483-4e68-a01a-5b0f024cce5c" (UID: "09a98cce-8483-4e68-a01a-5b0f024cce5c"). InnerVolumeSpecName "kube-api-access-r2zbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:25:02.379925 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:02.379903    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a98cce-8483-4e68-a01a-5b0f024cce5c-util" (OuterVolumeSpecName: "util") pod "09a98cce-8483-4e68-a01a-5b0f024cce5c" (UID: "09a98cce-8483-4e68-a01a-5b0f024cce5c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:25:02.476901 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:02.476860    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09a98cce-8483-4e68-a01a-5b0f024cce5c-bundle\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:25:02.476901 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:02.476896    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09a98cce-8483-4e68-a01a-5b0f024cce5c-util\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:25:02.476901 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:02.476906    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2zbx\" (UniqueName: \"kubernetes.io/projected/09a98cce-8483-4e68-a01a-5b0f024cce5c-kube-api-access-r2zbx\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:25:03.184202 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:03.184162    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv" event={"ID":"09a98cce-8483-4e68-a01a-5b0f024cce5c","Type":"ContainerDied","Data":"d799252d19e8e0d8cf10baa7ae1980fdbb10c4e9f707e808d919713228b0aeee"}
Sep 08 13:25:03.184407 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:03.184219    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d799252d19e8e0d8cf10baa7ae1980fdbb10c4e9f707e808d919713228b0aeee"
Sep 08 13:25:03.184407 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:03.184172    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ms4rv"
Sep 08 13:25:13.284696 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.284659    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-vgnf6"]
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.284964    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5928638b-1cf2-4486-a94f-182853c798d3" containerName="util"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.284975    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5928638b-1cf2-4486-a94f-182853c798d3" containerName="util"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.284984    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a98cce-8483-4e68-a01a-5b0f024cce5c" containerName="util"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.284989    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a98cce-8483-4e68-a01a-5b0f024cce5c" containerName="util"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.284996    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="233af0ef-37ac-41c9-a338-3f8fbd4814db" containerName="util"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285001    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="233af0ef-37ac-41c9-a338-3f8fbd4814db" containerName="util"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285006    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a98cce-8483-4e68-a01a-5b0f024cce5c" containerName="extract"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285012    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a98cce-8483-4e68-a01a-5b0f024cce5c" containerName="extract"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285018    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5928638b-1cf2-4486-a94f-182853c798d3" containerName="pull"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285023    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5928638b-1cf2-4486-a94f-182853c798d3" containerName="pull"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285033    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d6a572-7d35-4acd-beb8-edc776fa9033" containerName="extract"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285038    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d6a572-7d35-4acd-beb8-edc776fa9033" containerName="extract"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285045    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5928638b-1cf2-4486-a94f-182853c798d3" containerName="extract"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285051    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5928638b-1cf2-4486-a94f-182853c798d3" containerName="extract"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285056    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a98cce-8483-4e68-a01a-5b0f024cce5c" containerName="pull"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285060    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a98cce-8483-4e68-a01a-5b0f024cce5c" containerName="pull"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285067    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="233af0ef-37ac-41c9-a338-3f8fbd4814db" containerName="pull"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285071    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="233af0ef-37ac-41c9-a338-3f8fbd4814db" containerName="pull"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285077    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d6a572-7d35-4acd-beb8-edc776fa9033" containerName="util"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285082    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d6a572-7d35-4acd-beb8-edc776fa9033" containerName="util"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285092    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="233af0ef-37ac-41c9-a338-3f8fbd4814db" containerName="extract"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285098    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="233af0ef-37ac-41c9-a338-3f8fbd4814db" containerName="extract"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285103    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d6a572-7d35-4acd-beb8-edc776fa9033" containerName="pull"
Sep 08 13:25:13.285104 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285110    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d6a572-7d35-4acd-beb8-edc776fa9033" containerName="pull"
Sep 08 13:25:13.285865 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285156    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5928638b-1cf2-4486-a94f-182853c798d3" containerName="extract"
Sep 08 13:25:13.285865 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285165    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="233af0ef-37ac-41c9-a338-3f8fbd4814db" containerName="extract"
Sep 08 13:25:13.285865 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285171    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="09d6a572-7d35-4acd-beb8-edc776fa9033" containerName="extract"
Sep 08 13:25:13.285865 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.285177    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="09a98cce-8483-4e68-a01a-5b0f024cce5c" containerName="extract"
Sep 08 13:25:13.287459 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.287442    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-vgnf6"
Sep 08 13:25:13.294181 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.294162    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 08 13:25:13.296506 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.296486    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 08 13:25:13.307015 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.306993    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-449wh\""
Sep 08 13:25:13.308312 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.308276    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-vgnf6"]
Sep 08 13:25:13.367844 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.367813    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d039964a-035a-4e2f-bbc9-6a1aabf1d7be-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-vgnf6\" (UID: \"d039964a-035a-4e2f-bbc9-6a1aabf1d7be\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-vgnf6"
Sep 08 13:25:13.367988 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.367901    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8mhf\" (UniqueName: \"kubernetes.io/projected/d039964a-035a-4e2f-bbc9-6a1aabf1d7be-kube-api-access-k8mhf\") pod \"kuadrant-operator-controller-manager-66866f5fcc-vgnf6\" (UID: \"d039964a-035a-4e2f-bbc9-6a1aabf1d7be\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-vgnf6"
Sep 08 13:25:13.469050 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.469011    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d039964a-035a-4e2f-bbc9-6a1aabf1d7be-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-vgnf6\" (UID: \"d039964a-035a-4e2f-bbc9-6a1aabf1d7be\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-vgnf6"
Sep 08 13:25:13.469237 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.469102    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8mhf\" (UniqueName: \"kubernetes.io/projected/d039964a-035a-4e2f-bbc9-6a1aabf1d7be-kube-api-access-k8mhf\") pod \"kuadrant-operator-controller-manager-66866f5fcc-vgnf6\" (UID: \"d039964a-035a-4e2f-bbc9-6a1aabf1d7be\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-vgnf6"
Sep 08 13:25:13.469466 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.469443    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d039964a-035a-4e2f-bbc9-6a1aabf1d7be-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-vgnf6\" (UID: \"d039964a-035a-4e2f-bbc9-6a1aabf1d7be\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-vgnf6"
Sep 08 13:25:13.504402 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.504367    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8mhf\" (UniqueName: \"kubernetes.io/projected/d039964a-035a-4e2f-bbc9-6a1aabf1d7be-kube-api-access-k8mhf\") pod \"kuadrant-operator-controller-manager-66866f5fcc-vgnf6\" (UID: \"d039964a-035a-4e2f-bbc9-6a1aabf1d7be\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-vgnf6"
Sep 08 13:25:13.597312 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.597190    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-vgnf6"
Sep 08 13:25:13.882842 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:13.881536    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-vgnf6"]
Sep 08 13:25:13.887591 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:25:13.887557    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd039964a_035a_4e2f_bbc9_6a1aabf1d7be.slice/crio-221968cb958f3c88459bf77381635f094e4d28eeb28e189462a55be2ad48b928 WatchSource:0}: Error finding container 221968cb958f3c88459bf77381635f094e4d28eeb28e189462a55be2ad48b928: Status 404 returned error can't find the container with id 221968cb958f3c88459bf77381635f094e4d28eeb28e189462a55be2ad48b928
Sep 08 13:25:14.241130 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:14.241096    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-vgnf6" event={"ID":"d039964a-035a-4e2f-bbc9-6a1aabf1d7be","Type":"ContainerStarted","Data":"221968cb958f3c88459bf77381635f094e4d28eeb28e189462a55be2ad48b928"}
Sep 08 13:25:15.573783 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:15.573746    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-znxvr"]
Sep 08 13:25:15.576593 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:15.576575    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-znxvr"
Sep 08 13:25:15.580632 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:15.580607    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-pz9h7\""
Sep 08 13:25:15.592178 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:15.592145    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-znxvr"]
Sep 08 13:25:15.688128 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:15.688091    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4k8b\" (UniqueName: \"kubernetes.io/projected/b2dc2c9e-aec8-4eb7-a237-4ad984cb56be-kube-api-access-z4k8b\") pod \"authorino-operator-56fb6f6856-znxvr\" (UID: \"b2dc2c9e-aec8-4eb7-a237-4ad984cb56be\") " pod="kuadrant-system/authorino-operator-56fb6f6856-znxvr"
Sep 08 13:25:15.789697 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:15.789656    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4k8b\" (UniqueName: \"kubernetes.io/projected/b2dc2c9e-aec8-4eb7-a237-4ad984cb56be-kube-api-access-z4k8b\") pod \"authorino-operator-56fb6f6856-znxvr\" (UID: \"b2dc2c9e-aec8-4eb7-a237-4ad984cb56be\") " pod="kuadrant-system/authorino-operator-56fb6f6856-znxvr"
Sep 08 13:25:15.805605 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:15.805538    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4k8b\" (UniqueName: \"kubernetes.io/projected/b2dc2c9e-aec8-4eb7-a237-4ad984cb56be-kube-api-access-z4k8b\") pod \"authorino-operator-56fb6f6856-znxvr\" (UID: \"b2dc2c9e-aec8-4eb7-a237-4ad984cb56be\") " pod="kuadrant-system/authorino-operator-56fb6f6856-znxvr"
Sep 08 13:25:15.887606 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:15.887510    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-znxvr"
Sep 08 13:25:16.217794 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:16.217763    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-znxvr"]
Sep 08 13:25:16.254450 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:16.254405    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-znxvr" event={"ID":"b2dc2c9e-aec8-4eb7-a237-4ad984cb56be","Type":"ContainerStarted","Data":"ae667228fce3e2139dd1907123a053cfed342cd7b1b0d6caf5f4d18eda04ee6b"}
Sep 08 13:25:19.274534 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:19.274497    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-vgnf6" event={"ID":"d039964a-035a-4e2f-bbc9-6a1aabf1d7be","Type":"ContainerStarted","Data":"919eff94b767a4719a7504370fdc7b63d16d282b3d3adc9b088d6b2d896f4c95"}
Sep 08 13:25:19.275265 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:19.275240    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-vgnf6"
Sep 08 13:25:19.298871 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:19.298805    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-vgnf6" podStartSLOduration=1.327086491 podStartE2EDuration="6.298781925s" totalImagesPullingTime="4.971695434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:25:19.29678139 +0000 UTC m=+550.757426199" watchObservedRunningTime="2026-09-08 13:25:19.298781925 +0000 UTC m=+550.759426734"
Sep 08 13:25:20.280911 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:20.280872    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-znxvr" event={"ID":"b2dc2c9e-aec8-4eb7-a237-4ad984cb56be","Type":"ContainerStarted","Data":"566fb8e89d338b5a9ed0439561e7ae63f2015d4faf6174e2cca80d13fb552126"}
Sep 08 13:25:20.281327 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:20.281030    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-56fb6f6856-znxvr"
Sep 08 13:25:20.306589 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:20.306512    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-56fb6f6856-znxvr" podStartSLOduration=1.9600808870000002 podStartE2EDuration="5.306497359s" totalImagesPullingTime="3.346416472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:25:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:25:20.304808474 +0000 UTC m=+551.765453287" watchObservedRunningTime="2026-09-08 13:25:20.306497359 +0000 UTC m=+551.767142169"
Sep 08 13:25:31.288116 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:31.288074    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-56fb6f6856-znxvr"
Sep 08 13:25:31.288585 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:25:31.288198    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-vgnf6"
Sep 08 13:26:05.908209 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:05.908173    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-fn5cz"]
Sep 08 13:26:05.910691 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:05.910670    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-fn5cz"
Sep 08 13:26:05.912860 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:05.912837    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 08 13:26:05.912959 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:05.912901    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nlm5f\""
Sep 08 13:26:05.921965 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:05.921940    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-fn5cz"]
Sep 08 13:26:06.003204 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:06.003166    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-fn5cz"]
Sep 08 13:26:06.019904 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:06.019861    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vzgc\" (UniqueName: \"kubernetes.io/projected/05609044-9723-46e2-aa0d-32d4f07b6d1a-kube-api-access-2vzgc\") pod \"limitador-limitador-859c4d96d6-fn5cz\" (UID: \"05609044-9723-46e2-aa0d-32d4f07b6d1a\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-fn5cz"
Sep 08 13:26:06.020097 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:06.019989    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/05609044-9723-46e2-aa0d-32d4f07b6d1a-config-file\") pod \"limitador-limitador-859c4d96d6-fn5cz\" (UID: \"05609044-9723-46e2-aa0d-32d4f07b6d1a\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-fn5cz"
Sep 08 13:26:06.121538 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:06.121495    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/05609044-9723-46e2-aa0d-32d4f07b6d1a-config-file\") pod \"limitador-limitador-859c4d96d6-fn5cz\" (UID: \"05609044-9723-46e2-aa0d-32d4f07b6d1a\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-fn5cz"
Sep 08 13:26:06.121733 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:06.121602    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vzgc\" (UniqueName: \"kubernetes.io/projected/05609044-9723-46e2-aa0d-32d4f07b6d1a-kube-api-access-2vzgc\") pod \"limitador-limitador-859c4d96d6-fn5cz\" (UID: \"05609044-9723-46e2-aa0d-32d4f07b6d1a\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-fn5cz"
Sep 08 13:26:06.122596 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:06.122566    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/05609044-9723-46e2-aa0d-32d4f07b6d1a-config-file\") pod \"limitador-limitador-859c4d96d6-fn5cz\" (UID: \"05609044-9723-46e2-aa0d-32d4f07b6d1a\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-fn5cz"
Sep 08 13:26:06.130384 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:06.130348    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vzgc\" (UniqueName: \"kubernetes.io/projected/05609044-9723-46e2-aa0d-32d4f07b6d1a-kube-api-access-2vzgc\") pod \"limitador-limitador-859c4d96d6-fn5cz\" (UID: \"05609044-9723-46e2-aa0d-32d4f07b6d1a\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-fn5cz"
Sep 08 13:26:06.222428 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:06.222388    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-fn5cz"
Sep 08 13:26:06.497977 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:06.497951    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-fn5cz"]
Sep 08 13:26:06.500347 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:26:06.500315    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05609044_9723_46e2_aa0d_32d4f07b6d1a.slice/crio-1ab8a918f03112ed63efc474a81cc83d5ecd5d887ecb1f88b7c51712b45b2ec4 WatchSource:0}: Error finding container 1ab8a918f03112ed63efc474a81cc83d5ecd5d887ecb1f88b7c51712b45b2ec4: Status 404 returned error can't find the container with id 1ab8a918f03112ed63efc474a81cc83d5ecd5d887ecb1f88b7c51712b45b2ec4
Sep 08 13:26:06.513024 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:06.512986    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-fn5cz" event={"ID":"05609044-9723-46e2-aa0d-32d4f07b6d1a","Type":"ContainerStarted","Data":"1ab8a918f03112ed63efc474a81cc83d5ecd5d887ecb1f88b7c51712b45b2ec4"}
Sep 08 13:26:08.951215 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:08.951187    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 13:26:08.952185 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:08.952161    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 13:26:09.529569 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:09.529527    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-fn5cz" event={"ID":"05609044-9723-46e2-aa0d-32d4f07b6d1a","Type":"ContainerStarted","Data":"0a1d89b3146384810324169a85ff8db53e19998439a3639bab9e73fadf91d67b"}
Sep 08 13:26:09.529767 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:09.529682    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-fn5cz"
Sep 08 13:26:09.545664 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:09.545601    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-fn5cz" podStartSLOduration=1.6260132280000001 podStartE2EDuration="4.545580441s" totalImagesPullingTime="2.919567213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:26:09.544755144 +0000 UTC m=+601.005399954" watchObservedRunningTime="2026-09-08 13:26:09.545580441 +0000 UTC m=+601.006225252"
Sep 08 13:26:20.535811 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:20.535773    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-fn5cz"
Sep 08 13:26:23.449925 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:23.449890    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-fn5cz"]
Sep 08 13:26:23.450335 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:23.450080    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-fn5cz" podUID="05609044-9723-46e2-aa0d-32d4f07b6d1a" containerName="limitador" containerID="cri-o://0a1d89b3146384810324169a85ff8db53e19998439a3639bab9e73fadf91d67b" gracePeriod=30
Sep 08 13:26:23.991131 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:23.991105    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-fn5cz"
Sep 08 13:26:24.082344 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:24.082212    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vzgc\" (UniqueName: \"kubernetes.io/projected/05609044-9723-46e2-aa0d-32d4f07b6d1a-kube-api-access-2vzgc\") pod \"05609044-9723-46e2-aa0d-32d4f07b6d1a\" (UID: \"05609044-9723-46e2-aa0d-32d4f07b6d1a\") "
Sep 08 13:26:24.082502 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:24.082372    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/05609044-9723-46e2-aa0d-32d4f07b6d1a-config-file\") pod \"05609044-9723-46e2-aa0d-32d4f07b6d1a\" (UID: \"05609044-9723-46e2-aa0d-32d4f07b6d1a\") "
Sep 08 13:26:24.082739 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:24.082716    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05609044-9723-46e2-aa0d-32d4f07b6d1a-config-file" (OuterVolumeSpecName: "config-file") pod "05609044-9723-46e2-aa0d-32d4f07b6d1a" (UID: "05609044-9723-46e2-aa0d-32d4f07b6d1a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:26:24.084640 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:24.084611    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05609044-9723-46e2-aa0d-32d4f07b6d1a-kube-api-access-2vzgc" (OuterVolumeSpecName: "kube-api-access-2vzgc") pod "05609044-9723-46e2-aa0d-32d4f07b6d1a" (UID: "05609044-9723-46e2-aa0d-32d4f07b6d1a"). InnerVolumeSpecName "kube-api-access-2vzgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:26:24.183227 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:24.183190    2561 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/05609044-9723-46e2-aa0d-32d4f07b6d1a-config-file\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:26:24.183227 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:24.183220    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vzgc\" (UniqueName: \"kubernetes.io/projected/05609044-9723-46e2-aa0d-32d4f07b6d1a-kube-api-access-2vzgc\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:26:24.611943 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:24.611905    2561 generic.go:356] "Generic (PLEG): container finished" podID="05609044-9723-46e2-aa0d-32d4f07b6d1a" containerID="0a1d89b3146384810324169a85ff8db53e19998439a3639bab9e73fadf91d67b" exitCode=0
Sep 08 13:26:24.612548 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:24.611955    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-fn5cz" event={"ID":"05609044-9723-46e2-aa0d-32d4f07b6d1a","Type":"ContainerDied","Data":"0a1d89b3146384810324169a85ff8db53e19998439a3639bab9e73fadf91d67b"}
Sep 08 13:26:24.612548 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:24.611966    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-fn5cz"
Sep 08 13:26:24.612548 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:24.612022    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-fn5cz" event={"ID":"05609044-9723-46e2-aa0d-32d4f07b6d1a","Type":"ContainerDied","Data":"1ab8a918f03112ed63efc474a81cc83d5ecd5d887ecb1f88b7c51712b45b2ec4"}
Sep 08 13:26:24.612548 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:24.612044    2561 scope.go:117] "RemoveContainer" containerID="0a1d89b3146384810324169a85ff8db53e19998439a3639bab9e73fadf91d67b"
Sep 08 13:26:24.622532 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:24.622513    2561 scope.go:117] "RemoveContainer" containerID="0a1d89b3146384810324169a85ff8db53e19998439a3639bab9e73fadf91d67b"
Sep 08 13:26:24.622835 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:26:24.622816    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a1d89b3146384810324169a85ff8db53e19998439a3639bab9e73fadf91d67b\": container with ID starting with 0a1d89b3146384810324169a85ff8db53e19998439a3639bab9e73fadf91d67b not found: ID does not exist" containerID="0a1d89b3146384810324169a85ff8db53e19998439a3639bab9e73fadf91d67b"
Sep 08 13:26:24.622880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:24.622846    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1d89b3146384810324169a85ff8db53e19998439a3639bab9e73fadf91d67b"} err="failed to get container status \"0a1d89b3146384810324169a85ff8db53e19998439a3639bab9e73fadf91d67b\": rpc error: code = NotFound desc = could not find container \"0a1d89b3146384810324169a85ff8db53e19998439a3639bab9e73fadf91d67b\": container with ID starting with 0a1d89b3146384810324169a85ff8db53e19998439a3639bab9e73fadf91d67b not found: ID does not exist"
Sep 08 13:26:24.634369 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:24.634334    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-fn5cz"]
Sep 08 13:26:24.638228 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:24.638199    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-fn5cz"]
Sep 08 13:26:25.006209 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:25.006178    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05609044-9723-46e2-aa0d-32d4f07b6d1a" path="/var/lib/kubelet/pods/05609044-9723-46e2-aa0d-32d4f07b6d1a/volumes"
Sep 08 13:26:51.144719 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.144678    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"]
Sep 08 13:26:51.145175 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.145070    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05609044-9723-46e2-aa0d-32d4f07b6d1a" containerName="limitador"
Sep 08 13:26:51.145175 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.145084    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="05609044-9723-46e2-aa0d-32d4f07b6d1a" containerName="limitador"
Sep 08 13:26:51.145175 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.145146    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="05609044-9723-46e2-aa0d-32d4f07b6d1a" containerName="limitador"
Sep 08 13:26:51.149001 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.148983    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.162887 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.162855    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"]
Sep 08 13:26:51.207910 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.207876    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1def0316-87d7-4d12-b8ec-739a5dbe6f1a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-g49x5\" (UID: \"1def0316-87d7-4d12-b8ec-739a5dbe6f1a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.208083 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.207930    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1def0316-87d7-4d12-b8ec-739a5dbe6f1a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-g49x5\" (UID: \"1def0316-87d7-4d12-b8ec-739a5dbe6f1a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.208083 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.207950    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1def0316-87d7-4d12-b8ec-739a5dbe6f1a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-g49x5\" (UID: \"1def0316-87d7-4d12-b8ec-739a5dbe6f1a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.208083 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.207999    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1def0316-87d7-4d12-b8ec-739a5dbe6f1a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-g49x5\" (UID: \"1def0316-87d7-4d12-b8ec-739a5dbe6f1a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.208083 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.208039    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1def0316-87d7-4d12-b8ec-739a5dbe6f1a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-g49x5\" (UID: \"1def0316-87d7-4d12-b8ec-739a5dbe6f1a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.208228 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.208102    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1def0316-87d7-4d12-b8ec-739a5dbe6f1a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-g49x5\" (UID: \"1def0316-87d7-4d12-b8ec-739a5dbe6f1a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.208228 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.208133    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmzq7\" (UniqueName: \"kubernetes.io/projected/1def0316-87d7-4d12-b8ec-739a5dbe6f1a-kube-api-access-tmzq7\") pod \"istiod-openshift-gateway-5677ccb49b-g49x5\" (UID: \"1def0316-87d7-4d12-b8ec-739a5dbe6f1a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.308705 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.308662    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1def0316-87d7-4d12-b8ec-739a5dbe6f1a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-g49x5\" (UID: \"1def0316-87d7-4d12-b8ec-739a5dbe6f1a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.308705 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.308704    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1def0316-87d7-4d12-b8ec-739a5dbe6f1a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-g49x5\" (UID: \"1def0316-87d7-4d12-b8ec-739a5dbe6f1a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.308937 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.308729    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1def0316-87d7-4d12-b8ec-739a5dbe6f1a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-g49x5\" (UID: \"1def0316-87d7-4d12-b8ec-739a5dbe6f1a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.308937 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.308753    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1def0316-87d7-4d12-b8ec-739a5dbe6f1a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-g49x5\" (UID: \"1def0316-87d7-4d12-b8ec-739a5dbe6f1a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.308937 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.308805    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1def0316-87d7-4d12-b8ec-739a5dbe6f1a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-g49x5\" (UID: \"1def0316-87d7-4d12-b8ec-739a5dbe6f1a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.308937 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.308838    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmzq7\" (UniqueName: \"kubernetes.io/projected/1def0316-87d7-4d12-b8ec-739a5dbe6f1a-kube-api-access-tmzq7\") pod \"istiod-openshift-gateway-5677ccb49b-g49x5\" (UID: \"1def0316-87d7-4d12-b8ec-739a5dbe6f1a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.308937 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.308882    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1def0316-87d7-4d12-b8ec-739a5dbe6f1a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-g49x5\" (UID: \"1def0316-87d7-4d12-b8ec-739a5dbe6f1a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.309511 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.309487    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1def0316-87d7-4d12-b8ec-739a5dbe6f1a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-g49x5\" (UID: \"1def0316-87d7-4d12-b8ec-739a5dbe6f1a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.311185 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.311164    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1def0316-87d7-4d12-b8ec-739a5dbe6f1a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-g49x5\" (UID: \"1def0316-87d7-4d12-b8ec-739a5dbe6f1a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.311374 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.311347    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1def0316-87d7-4d12-b8ec-739a5dbe6f1a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-g49x5\" (UID: \"1def0316-87d7-4d12-b8ec-739a5dbe6f1a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.311615 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.311599    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1def0316-87d7-4d12-b8ec-739a5dbe6f1a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-g49x5\" (UID: \"1def0316-87d7-4d12-b8ec-739a5dbe6f1a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.311689 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.311673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1def0316-87d7-4d12-b8ec-739a5dbe6f1a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-g49x5\" (UID: \"1def0316-87d7-4d12-b8ec-739a5dbe6f1a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.317738 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.317708    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1def0316-87d7-4d12-b8ec-739a5dbe6f1a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-g49x5\" (UID: \"1def0316-87d7-4d12-b8ec-739a5dbe6f1a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.317939 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.317921    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmzq7\" (UniqueName: \"kubernetes.io/projected/1def0316-87d7-4d12-b8ec-739a5dbe6f1a-kube-api-access-tmzq7\") pod \"istiod-openshift-gateway-5677ccb49b-g49x5\" (UID: \"1def0316-87d7-4d12-b8ec-739a5dbe6f1a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.458877 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.458847    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:51.741047 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.741017    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"]
Sep 08 13:26:51.742839 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:26:51.742808    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1def0316_87d7_4d12_b8ec_739a5dbe6f1a.slice/crio-08e0dcd12c2b921d945f649da7d4508d730ffd452d60dbb73c0d85b8cf2200d4 WatchSource:0}: Error finding container 08e0dcd12c2b921d945f649da7d4508d730ffd452d60dbb73c0d85b8cf2200d4: Status 404 returned error can't find the container with id 08e0dcd12c2b921d945f649da7d4508d730ffd452d60dbb73c0d85b8cf2200d4
Sep 08 13:26:51.748199 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.748156    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 13:26:51.748374 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.748242    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 13:26:51.753800 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:51.753771    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5" event={"ID":"1def0316-87d7-4d12-b8ec-739a5dbe6f1a","Type":"ContainerStarted","Data":"08e0dcd12c2b921d945f649da7d4508d730ffd452d60dbb73c0d85b8cf2200d4"}
Sep 08 13:26:52.760166 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:52.760133    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5" event={"ID":"1def0316-87d7-4d12-b8ec-739a5dbe6f1a","Type":"ContainerStarted","Data":"c95adc61970f6accc7aaad5bc9fa91e1552028515f48257e598d3396e4238a75"}
Sep 08 13:26:52.760749 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:52.760187    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:52.762351 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:52.762328    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5"
Sep 08 13:26:52.784125 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:52.784072    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g49x5" podStartSLOduration=1.784053152 podStartE2EDuration="1.784053152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:26:52.782035297 +0000 UTC m=+644.242680111" watchObservedRunningTime="2026-09-08 13:26:52.784053152 +0000 UTC m=+644.244697962"
Sep 08 13:26:52.840510 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:52.840474    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"]
Sep 08 13:26:52.840730 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:52.840707    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb" podUID="be5770ee-8459-4a0b-840e-a93aeaf3603a" containerName="discovery" containerID="cri-o://d3d46048481af486a41ef8c67ca2dd04d4636333776d66705a0ee9b0a552f839" gracePeriod=30
Sep 08 13:26:53.125398 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.125369    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:26:53.227558 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.227521    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-csr-dns-cert\") pod \"be5770ee-8459-4a0b-840e-a93aeaf3603a\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") "
Sep 08 13:26:53.227749 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.227575    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hd6vj\" (UniqueName: \"kubernetes.io/projected/be5770ee-8459-4a0b-840e-a93aeaf3603a-kube-api-access-hd6vj\") pod \"be5770ee-8459-4a0b-840e-a93aeaf3603a\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") "
Sep 08 13:26:53.227749 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.227601    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-token\") pod \"be5770ee-8459-4a0b-840e-a93aeaf3603a\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") "
Sep 08 13:26:53.227749 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.227629    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-csr-ca-configmap\") pod \"be5770ee-8459-4a0b-840e-a93aeaf3603a\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") "
Sep 08 13:26:53.227749 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.227651    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5770ee-8459-4a0b-840e-a93aeaf3603a-local-certs\") pod \"be5770ee-8459-4a0b-840e-a93aeaf3603a\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") "
Sep 08 13:26:53.227749 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.227694    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be5770ee-8459-4a0b-840e-a93aeaf3603a-cacerts\") pod \"be5770ee-8459-4a0b-840e-a93aeaf3603a\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") "
Sep 08 13:26:53.227749 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.227741    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-kubeconfig\") pod \"be5770ee-8459-4a0b-840e-a93aeaf3603a\" (UID: \"be5770ee-8459-4a0b-840e-a93aeaf3603a\") "
Sep 08 13:26:53.228035 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.228011    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "be5770ee-8459-4a0b-840e-a93aeaf3603a" (UID: "be5770ee-8459-4a0b-840e-a93aeaf3603a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:26:53.230201 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.230134    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5770ee-8459-4a0b-840e-a93aeaf3603a-local-certs" (OuterVolumeSpecName: "local-certs") pod "be5770ee-8459-4a0b-840e-a93aeaf3603a" (UID: "be5770ee-8459-4a0b-840e-a93aeaf3603a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:26:53.230545 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.230518    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "be5770ee-8459-4a0b-840e-a93aeaf3603a" (UID: "be5770ee-8459-4a0b-840e-a93aeaf3603a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:26:53.230733 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.230704    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-token" (OuterVolumeSpecName: "istio-token") pod "be5770ee-8459-4a0b-840e-a93aeaf3603a" (UID: "be5770ee-8459-4a0b-840e-a93aeaf3603a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:26:53.230812 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.230524    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be5770ee-8459-4a0b-840e-a93aeaf3603a-cacerts" (OuterVolumeSpecName: "cacerts") pod "be5770ee-8459-4a0b-840e-a93aeaf3603a" (UID: "be5770ee-8459-4a0b-840e-a93aeaf3603a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:26:53.230812 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.230582    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be5770ee-8459-4a0b-840e-a93aeaf3603a-kube-api-access-hd6vj" (OuterVolumeSpecName: "kube-api-access-hd6vj") pod "be5770ee-8459-4a0b-840e-a93aeaf3603a" (UID: "be5770ee-8459-4a0b-840e-a93aeaf3603a"). InnerVolumeSpecName "kube-api-access-hd6vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:26:53.230812 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.230586    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "be5770ee-8459-4a0b-840e-a93aeaf3603a" (UID: "be5770ee-8459-4a0b-840e-a93aeaf3603a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:26:53.329282 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.329174    2561 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be5770ee-8459-4a0b-840e-a93aeaf3603a-cacerts\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:26:53.329282 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.329210    2561 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-kubeconfig\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:26:53.329282 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.329228    2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-csr-dns-cert\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:26:53.329282 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.329241    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd6vj\" (UniqueName: \"kubernetes.io/projected/be5770ee-8459-4a0b-840e-a93aeaf3603a-kube-api-access-hd6vj\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:26:53.329282 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.329252    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-token\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:26:53.329282 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.329262    2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be5770ee-8459-4a0b-840e-a93aeaf3603a-istio-csr-ca-configmap\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:26:53.329282 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.329277    2561 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5770ee-8459-4a0b-840e-a93aeaf3603a-local-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:26:53.766586 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.766555    2561 generic.go:356] "Generic (PLEG): container finished" podID="be5770ee-8459-4a0b-840e-a93aeaf3603a" containerID="d3d46048481af486a41ef8c67ca2dd04d4636333776d66705a0ee9b0a552f839" exitCode=0
Sep 08 13:26:53.767011 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.766616    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"
Sep 08 13:26:53.767011 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.766636    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb" event={"ID":"be5770ee-8459-4a0b-840e-a93aeaf3603a","Type":"ContainerDied","Data":"d3d46048481af486a41ef8c67ca2dd04d4636333776d66705a0ee9b0a552f839"}
Sep 08 13:26:53.767011 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.766704    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb" event={"ID":"be5770ee-8459-4a0b-840e-a93aeaf3603a","Type":"ContainerDied","Data":"3d12ffeaa0f949465a17c96fffedd7648b9c55b1d3dc1c312af339bab46d3437"}
Sep 08 13:26:53.767011 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.766724    2561 scope.go:117] "RemoveContainer" containerID="d3d46048481af486a41ef8c67ca2dd04d4636333776d66705a0ee9b0a552f839"
Sep 08 13:26:53.776065 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.776045    2561 scope.go:117] "RemoveContainer" containerID="d3d46048481af486a41ef8c67ca2dd04d4636333776d66705a0ee9b0a552f839"
Sep 08 13:26:53.776483 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:26:53.776464    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3d46048481af486a41ef8c67ca2dd04d4636333776d66705a0ee9b0a552f839\": container with ID starting with d3d46048481af486a41ef8c67ca2dd04d4636333776d66705a0ee9b0a552f839 not found: ID does not exist" containerID="d3d46048481af486a41ef8c67ca2dd04d4636333776d66705a0ee9b0a552f839"
Sep 08 13:26:53.776527 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.776492    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3d46048481af486a41ef8c67ca2dd04d4636333776d66705a0ee9b0a552f839"} err="failed to get container status \"d3d46048481af486a41ef8c67ca2dd04d4636333776d66705a0ee9b0a552f839\": rpc error: code = NotFound desc = could not find container \"d3d46048481af486a41ef8c67ca2dd04d4636333776d66705a0ee9b0a552f839\": container with ID starting with d3d46048481af486a41ef8c67ca2dd04d4636333776d66705a0ee9b0a552f839 not found: ID does not exist"
Sep 08 13:26:53.788422 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.788388    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"]
Sep 08 13:26:53.792571 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:53.792542    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5gnb"]
Sep 08 13:26:55.006685 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:26:55.006646    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be5770ee-8459-4a0b-840e-a93aeaf3603a" path="/var/lib/kubelet/pods/be5770ee-8459-4a0b-840e-a93aeaf3603a/volumes"
Sep 08 13:27:03.180865 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.180825    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-575d6764f4-8ng96"]
Sep 08 13:27:03.181254 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.181147    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be5770ee-8459-4a0b-840e-a93aeaf3603a" containerName="discovery"
Sep 08 13:27:03.181254 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.181159    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="be5770ee-8459-4a0b-840e-a93aeaf3603a" containerName="discovery"
Sep 08 13:27:03.181254 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.181225    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="be5770ee-8459-4a0b-840e-a93aeaf3603a" containerName="discovery"
Sep 08 13:27:03.183926 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.183908    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-575d6764f4-8ng96"
Sep 08 13:27:03.186402 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.186371    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-bcjq5\""
Sep 08 13:27:03.186953 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.186931    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 08 13:27:03.187083 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.186963    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 08 13:27:03.187083 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.186936    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 08 13:27:03.195439 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.195414    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-575d6764f4-8ng96"]
Sep 08 13:27:03.199804 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.199781    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-945498d87-2sh5b"]
Sep 08 13:27:03.203009 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.202987    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-945498d87-2sh5b"
Sep 08 13:27:03.205342 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.205319    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-p9nl6\""
Sep 08 13:27:03.205547 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.205351    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 08 13:27:03.220967 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.220938    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-945498d87-2sh5b"]
Sep 08 13:27:03.326438 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.326401    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdr5s\" (UniqueName: \"kubernetes.io/projected/faaa3499-807c-436d-a97b-ae134b995040-kube-api-access-tdr5s\") pod \"llmisvc-controller-manager-945498d87-2sh5b\" (UID: \"faaa3499-807c-436d-a97b-ae134b995040\") " pod="kserve/llmisvc-controller-manager-945498d87-2sh5b"
Sep 08 13:27:03.326626 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.326476    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/faaa3499-807c-436d-a97b-ae134b995040-cert\") pod \"llmisvc-controller-manager-945498d87-2sh5b\" (UID: \"faaa3499-807c-436d-a97b-ae134b995040\") " pod="kserve/llmisvc-controller-manager-945498d87-2sh5b"
Sep 08 13:27:03.326626 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.326494    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3b2cb10-8a6a-4d09-b0b9-e48c600daad1-cert\") pod \"kserve-controller-manager-575d6764f4-8ng96\" (UID: \"f3b2cb10-8a6a-4d09-b0b9-e48c600daad1\") " pod="kserve/kserve-controller-manager-575d6764f4-8ng96"
Sep 08 13:27:03.326626 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.326532    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvj7d\" (UniqueName: \"kubernetes.io/projected/f3b2cb10-8a6a-4d09-b0b9-e48c600daad1-kube-api-access-zvj7d\") pod \"kserve-controller-manager-575d6764f4-8ng96\" (UID: \"f3b2cb10-8a6a-4d09-b0b9-e48c600daad1\") " pod="kserve/kserve-controller-manager-575d6764f4-8ng96"
Sep 08 13:27:03.427280 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.427245    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvj7d\" (UniqueName: \"kubernetes.io/projected/f3b2cb10-8a6a-4d09-b0b9-e48c600daad1-kube-api-access-zvj7d\") pod \"kserve-controller-manager-575d6764f4-8ng96\" (UID: \"f3b2cb10-8a6a-4d09-b0b9-e48c600daad1\") " pod="kserve/kserve-controller-manager-575d6764f4-8ng96"
Sep 08 13:27:03.427459 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.427308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdr5s\" (UniqueName: \"kubernetes.io/projected/faaa3499-807c-436d-a97b-ae134b995040-kube-api-access-tdr5s\") pod \"llmisvc-controller-manager-945498d87-2sh5b\" (UID: \"faaa3499-807c-436d-a97b-ae134b995040\") " pod="kserve/llmisvc-controller-manager-945498d87-2sh5b"
Sep 08 13:27:03.427459 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.427370    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/faaa3499-807c-436d-a97b-ae134b995040-cert\") pod \"llmisvc-controller-manager-945498d87-2sh5b\" (UID: \"faaa3499-807c-436d-a97b-ae134b995040\") " pod="kserve/llmisvc-controller-manager-945498d87-2sh5b"
Sep 08 13:27:03.427459 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.427388    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3b2cb10-8a6a-4d09-b0b9-e48c600daad1-cert\") pod \"kserve-controller-manager-575d6764f4-8ng96\" (UID: \"f3b2cb10-8a6a-4d09-b0b9-e48c600daad1\") " pod="kserve/kserve-controller-manager-575d6764f4-8ng96"
Sep 08 13:27:03.427583 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:27:03.427490    2561 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 08 13:27:03.427583 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:27:03.427567    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faaa3499-807c-436d-a97b-ae134b995040-cert podName:faaa3499-807c-436d-a97b-ae134b995040 nodeName:}" failed. No retries permitted until 2026-09-08 13:27:03.927546772 +0000 UTC m=+655.388191564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/faaa3499-807c-436d-a97b-ae134b995040-cert") pod "llmisvc-controller-manager-945498d87-2sh5b" (UID: "faaa3499-807c-436d-a97b-ae134b995040") : secret "llmisvc-webhook-server-cert" not found
Sep 08 13:27:03.429811 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.429792    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3b2cb10-8a6a-4d09-b0b9-e48c600daad1-cert\") pod \"kserve-controller-manager-575d6764f4-8ng96\" (UID: \"f3b2cb10-8a6a-4d09-b0b9-e48c600daad1\") " pod="kserve/kserve-controller-manager-575d6764f4-8ng96"
Sep 08 13:27:03.435802 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.435779    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvj7d\" (UniqueName: \"kubernetes.io/projected/f3b2cb10-8a6a-4d09-b0b9-e48c600daad1-kube-api-access-zvj7d\") pod \"kserve-controller-manager-575d6764f4-8ng96\" (UID: \"f3b2cb10-8a6a-4d09-b0b9-e48c600daad1\") " pod="kserve/kserve-controller-manager-575d6764f4-8ng96"
Sep 08 13:27:03.435802 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.435796    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdr5s\" (UniqueName: \"kubernetes.io/projected/faaa3499-807c-436d-a97b-ae134b995040-kube-api-access-tdr5s\") pod \"llmisvc-controller-manager-945498d87-2sh5b\" (UID: \"faaa3499-807c-436d-a97b-ae134b995040\") " pod="kserve/llmisvc-controller-manager-945498d87-2sh5b"
Sep 08 13:27:03.498210 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.498176    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-575d6764f4-8ng96"
Sep 08 13:27:03.777545 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.776390    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-575d6764f4-8ng96"]
Sep 08 13:27:03.818949 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.818911    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-575d6764f4-8ng96" event={"ID":"f3b2cb10-8a6a-4d09-b0b9-e48c600daad1","Type":"ContainerStarted","Data":"d11645c7455c3aa16fba0cd948640ddacc59f26760a909c6e35894b9c224e031"}
Sep 08 13:27:03.932168 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.932120    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/faaa3499-807c-436d-a97b-ae134b995040-cert\") pod \"llmisvc-controller-manager-945498d87-2sh5b\" (UID: \"faaa3499-807c-436d-a97b-ae134b995040\") " pod="kserve/llmisvc-controller-manager-945498d87-2sh5b"
Sep 08 13:27:03.934626 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:03.934588    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/faaa3499-807c-436d-a97b-ae134b995040-cert\") pod \"llmisvc-controller-manager-945498d87-2sh5b\" (UID: \"faaa3499-807c-436d-a97b-ae134b995040\") " pod="kserve/llmisvc-controller-manager-945498d87-2sh5b"
Sep 08 13:27:04.114541 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:04.114457    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-945498d87-2sh5b"
Sep 08 13:27:04.386316 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:04.386260    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-945498d87-2sh5b"]
Sep 08 13:27:04.387886 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:27:04.387862    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfaaa3499_807c_436d_a97b_ae134b995040.slice/crio-4abf9d2527a7593eec3f6666ea0434a397d256189a4f0b2b7190adc3854b9cc2 WatchSource:0}: Error finding container 4abf9d2527a7593eec3f6666ea0434a397d256189a4f0b2b7190adc3854b9cc2: Status 404 returned error can't find the container with id 4abf9d2527a7593eec3f6666ea0434a397d256189a4f0b2b7190adc3854b9cc2
Sep 08 13:27:04.825641 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:04.825608    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-945498d87-2sh5b" event={"ID":"faaa3499-807c-436d-a97b-ae134b995040","Type":"ContainerStarted","Data":"4abf9d2527a7593eec3f6666ea0434a397d256189a4f0b2b7190adc3854b9cc2"}
Sep 08 13:27:07.847694 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:07.847643    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-945498d87-2sh5b" event={"ID":"faaa3499-807c-436d-a97b-ae134b995040","Type":"ContainerStarted","Data":"662a9241359bf143dbe9d7f655d771c1e37c103c95b3dafab7937a0aeaa38fe6"}
Sep 08 13:27:07.848178 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:07.847791    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-945498d87-2sh5b"
Sep 08 13:27:07.864508 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:07.864448    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-945498d87-2sh5b" podStartSLOduration=1.9705975329999998 podStartE2EDuration="4.864430169s" totalImagesPullingTime="2.893832636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:27:07.863241981 +0000 UTC m=+659.323886790" watchObservedRunningTime="2026-09-08 13:27:07.864430169 +0000 UTC m=+659.325074978"
Sep 08 13:27:08.854259 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:08.854223    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-575d6764f4-8ng96" event={"ID":"f3b2cb10-8a6a-4d09-b0b9-e48c600daad1","Type":"ContainerStarted","Data":"bd8affd9dc7e2d3e379316c806b4d05ce1c0641c131b4a972c7aa0d29319d53d"}
Sep 08 13:27:08.854728 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:08.854341    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-575d6764f4-8ng96"
Sep 08 13:27:08.876186 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:08.871214    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-575d6764f4-8ng96" podStartSLOduration=1.479238195 podStartE2EDuration="5.871191253s" totalImagesPullingTime="4.391953058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:27:08.870163071 +0000 UTC m=+660.330807882" watchObservedRunningTime="2026-09-08 13:27:08.871191253 +0000 UTC m=+660.331836065"
Sep 08 13:27:38.856962 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:38.856927    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-945498d87-2sh5b"
Sep 08 13:27:39.865065 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:39.865024    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-575d6764f4-8ng96"
Sep 08 13:27:46.777421 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:46.777380    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-575d6764f4-8ng96"]
Sep 08 13:27:46.777825 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:46.777625    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-575d6764f4-8ng96" podUID="f3b2cb10-8a6a-4d09-b0b9-e48c600daad1" containerName="manager" containerID="cri-o://bd8affd9dc7e2d3e379316c806b4d05ce1c0641c131b4a972c7aa0d29319d53d" gracePeriod=10
Sep 08 13:27:46.813319 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:46.813268    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-575d6764f4-tw7r2"]
Sep 08 13:27:46.815737 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:46.815719    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-575d6764f4-tw7r2"
Sep 08 13:27:46.824486 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:46.824457    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-575d6764f4-tw7r2"]
Sep 08 13:27:46.911725 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:46.911686    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxktk\" (UniqueName: \"kubernetes.io/projected/c713585e-70c3-4be8-a1d5-91b521dc205a-kube-api-access-nxktk\") pod \"kserve-controller-manager-575d6764f4-tw7r2\" (UID: \"c713585e-70c3-4be8-a1d5-91b521dc205a\") " pod="kserve/kserve-controller-manager-575d6764f4-tw7r2"
Sep 08 13:27:46.911901 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:46.911742    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c713585e-70c3-4be8-a1d5-91b521dc205a-cert\") pod \"kserve-controller-manager-575d6764f4-tw7r2\" (UID: \"c713585e-70c3-4be8-a1d5-91b521dc205a\") " pod="kserve/kserve-controller-manager-575d6764f4-tw7r2"
Sep 08 13:27:47.012355 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.012326    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxktk\" (UniqueName: \"kubernetes.io/projected/c713585e-70c3-4be8-a1d5-91b521dc205a-kube-api-access-nxktk\") pod \"kserve-controller-manager-575d6764f4-tw7r2\" (UID: \"c713585e-70c3-4be8-a1d5-91b521dc205a\") " pod="kserve/kserve-controller-manager-575d6764f4-tw7r2"
Sep 08 13:27:47.012520 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.012379    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c713585e-70c3-4be8-a1d5-91b521dc205a-cert\") pod \"kserve-controller-manager-575d6764f4-tw7r2\" (UID: \"c713585e-70c3-4be8-a1d5-91b521dc205a\") " pod="kserve/kserve-controller-manager-575d6764f4-tw7r2"
Sep 08 13:27:47.014450 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.014421    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-575d6764f4-8ng96"
Sep 08 13:27:47.014731 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.014713    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c713585e-70c3-4be8-a1d5-91b521dc205a-cert\") pod \"kserve-controller-manager-575d6764f4-tw7r2\" (UID: \"c713585e-70c3-4be8-a1d5-91b521dc205a\") " pod="kserve/kserve-controller-manager-575d6764f4-tw7r2"
Sep 08 13:27:47.023989 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.023967    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxktk\" (UniqueName: \"kubernetes.io/projected/c713585e-70c3-4be8-a1d5-91b521dc205a-kube-api-access-nxktk\") pod \"kserve-controller-manager-575d6764f4-tw7r2\" (UID: \"c713585e-70c3-4be8-a1d5-91b521dc205a\") " pod="kserve/kserve-controller-manager-575d6764f4-tw7r2"
Sep 08 13:27:47.053491 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.053413    2561 generic.go:356] "Generic (PLEG): container finished" podID="f3b2cb10-8a6a-4d09-b0b9-e48c600daad1" containerID="bd8affd9dc7e2d3e379316c806b4d05ce1c0641c131b4a972c7aa0d29319d53d" exitCode=0
Sep 08 13:27:47.053491 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.053460    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-575d6764f4-8ng96"
Sep 08 13:27:47.053714 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.053494    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-575d6764f4-8ng96" event={"ID":"f3b2cb10-8a6a-4d09-b0b9-e48c600daad1","Type":"ContainerDied","Data":"bd8affd9dc7e2d3e379316c806b4d05ce1c0641c131b4a972c7aa0d29319d53d"}
Sep 08 13:27:47.053714 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.053549    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-575d6764f4-8ng96" event={"ID":"f3b2cb10-8a6a-4d09-b0b9-e48c600daad1","Type":"ContainerDied","Data":"d11645c7455c3aa16fba0cd948640ddacc59f26760a909c6e35894b9c224e031"}
Sep 08 13:27:47.053714 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.053579    2561 scope.go:117] "RemoveContainer" containerID="bd8affd9dc7e2d3e379316c806b4d05ce1c0641c131b4a972c7aa0d29319d53d"
Sep 08 13:27:47.063653 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.063634    2561 scope.go:117] "RemoveContainer" containerID="bd8affd9dc7e2d3e379316c806b4d05ce1c0641c131b4a972c7aa0d29319d53d"
Sep 08 13:27:47.063973 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:27:47.063950    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd8affd9dc7e2d3e379316c806b4d05ce1c0641c131b4a972c7aa0d29319d53d\": container with ID starting with bd8affd9dc7e2d3e379316c806b4d05ce1c0641c131b4a972c7aa0d29319d53d not found: ID does not exist" containerID="bd8affd9dc7e2d3e379316c806b4d05ce1c0641c131b4a972c7aa0d29319d53d"
Sep 08 13:27:47.064068 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.063978    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd8affd9dc7e2d3e379316c806b4d05ce1c0641c131b4a972c7aa0d29319d53d"} err="failed to get container status \"bd8affd9dc7e2d3e379316c806b4d05ce1c0641c131b4a972c7aa0d29319d53d\": rpc error: code = NotFound desc = could not find container \"bd8affd9dc7e2d3e379316c806b4d05ce1c0641c131b4a972c7aa0d29319d53d\": container with ID starting with bd8affd9dc7e2d3e379316c806b4d05ce1c0641c131b4a972c7aa0d29319d53d not found: ID does not exist"
Sep 08 13:27:47.113485 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.113455    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvj7d\" (UniqueName: \"kubernetes.io/projected/f3b2cb10-8a6a-4d09-b0b9-e48c600daad1-kube-api-access-zvj7d\") pod \"f3b2cb10-8a6a-4d09-b0b9-e48c600daad1\" (UID: \"f3b2cb10-8a6a-4d09-b0b9-e48c600daad1\") "
Sep 08 13:27:47.113650 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.113578    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3b2cb10-8a6a-4d09-b0b9-e48c600daad1-cert\") pod \"f3b2cb10-8a6a-4d09-b0b9-e48c600daad1\" (UID: \"f3b2cb10-8a6a-4d09-b0b9-e48c600daad1\") "
Sep 08 13:27:47.115724 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.115692    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b2cb10-8a6a-4d09-b0b9-e48c600daad1-cert" (OuterVolumeSpecName: "cert") pod "f3b2cb10-8a6a-4d09-b0b9-e48c600daad1" (UID: "f3b2cb10-8a6a-4d09-b0b9-e48c600daad1"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:27:47.115724 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.115711    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b2cb10-8a6a-4d09-b0b9-e48c600daad1-kube-api-access-zvj7d" (OuterVolumeSpecName: "kube-api-access-zvj7d") pod "f3b2cb10-8a6a-4d09-b0b9-e48c600daad1" (UID: "f3b2cb10-8a6a-4d09-b0b9-e48c600daad1"). InnerVolumeSpecName "kube-api-access-zvj7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:27:47.155594 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.155561    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-575d6764f4-tw7r2"
Sep 08 13:27:47.214760 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.214728    2561 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3b2cb10-8a6a-4d09-b0b9-e48c600daad1-cert\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:27:47.214760 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.214760    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvj7d\" (UniqueName: \"kubernetes.io/projected/f3b2cb10-8a6a-4d09-b0b9-e48c600daad1-kube-api-access-zvj7d\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:27:47.377043 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.377010    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-575d6764f4-8ng96"]
Sep 08 13:27:47.382466 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.382437    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-575d6764f4-8ng96"]
Sep 08 13:27:47.426525 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.426502    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-575d6764f4-tw7r2"]
Sep 08 13:27:47.428397 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:27:47.428369    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc713585e_70c3_4be8_a1d5_91b521dc205a.slice/crio-d39565e25c27b8b18cc3733d16b0b61f7f037f37a84d3e65626b21a159cca5e2 WatchSource:0}: Error finding container d39565e25c27b8b18cc3733d16b0b61f7f037f37a84d3e65626b21a159cca5e2: Status 404 returned error can't find the container with id d39565e25c27b8b18cc3733d16b0b61f7f037f37a84d3e65626b21a159cca5e2
Sep 08 13:27:47.429723 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:47.429708    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:27:48.061150 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:48.061060    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-575d6764f4-tw7r2" event={"ID":"c713585e-70c3-4be8-a1d5-91b521dc205a","Type":"ContainerStarted","Data":"d39565e25c27b8b18cc3733d16b0b61f7f037f37a84d3e65626b21a159cca5e2"}
Sep 08 13:27:49.013643 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:49.013608    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3b2cb10-8a6a-4d09-b0b9-e48c600daad1" path="/var/lib/kubelet/pods/f3b2cb10-8a6a-4d09-b0b9-e48c600daad1/volumes"
Sep 08 13:27:49.068411 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:49.068377    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-575d6764f4-tw7r2" event={"ID":"c713585e-70c3-4be8-a1d5-91b521dc205a","Type":"ContainerStarted","Data":"200cf672d58d3d622166273a9a6a8e64d8022e9fed181b8a52fb7e9fd4a9cafc"}
Sep 08 13:27:49.068794 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:49.068487    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-575d6764f4-tw7r2"
Sep 08 13:27:49.086887 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:27:49.086832    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-575d6764f4-tw7r2" podStartSLOduration=2.377402364 podStartE2EDuration="3.086815418s" totalImagesPullingTime="709.413054ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:27:49.08568437 +0000 UTC m=+700.546329182" watchObservedRunningTime="2026-09-08 13:27:49.086815418 +0000 UTC m=+700.547460286"
Sep 08 13:28:20.078948 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:20.078912    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-575d6764f4-tw7r2"
Sep 08 13:28:21.581544 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:21.581469    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-9hzrh"]
Sep 08 13:28:21.582543 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:21.582090    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3b2cb10-8a6a-4d09-b0b9-e48c600daad1" containerName="manager"
Sep 08 13:28:21.582543 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:21.582111    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3b2cb10-8a6a-4d09-b0b9-e48c600daad1" containerName="manager"
Sep 08 13:28:21.582543 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:21.582216    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3b2cb10-8a6a-4d09-b0b9-e48c600daad1" containerName="manager"
Sep 08 13:28:21.585463 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:21.585443    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-9hzrh"
Sep 08 13:28:21.588490 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:21.588470    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 08 13:28:21.588615 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:21.588569    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-b4m8r\""
Sep 08 13:28:21.601769 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:21.601744    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-9hzrh"]
Sep 08 13:28:21.614228 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:21.614188    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqv9p\" (UniqueName: \"kubernetes.io/projected/511f6abd-9084-449c-a2fb-4c4035463892-kube-api-access-qqv9p\") pod \"model-serving-api-7b954ffc95-9hzrh\" (UID: \"511f6abd-9084-449c-a2fb-4c4035463892\") " pod="kserve/model-serving-api-7b954ffc95-9hzrh"
Sep 08 13:28:21.614410 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:21.614247    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/511f6abd-9084-449c-a2fb-4c4035463892-tls-certs\") pod \"model-serving-api-7b954ffc95-9hzrh\" (UID: \"511f6abd-9084-449c-a2fb-4c4035463892\") " pod="kserve/model-serving-api-7b954ffc95-9hzrh"
Sep 08 13:28:21.715132 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:21.715089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqv9p\" (UniqueName: \"kubernetes.io/projected/511f6abd-9084-449c-a2fb-4c4035463892-kube-api-access-qqv9p\") pod \"model-serving-api-7b954ffc95-9hzrh\" (UID: \"511f6abd-9084-449c-a2fb-4c4035463892\") " pod="kserve/model-serving-api-7b954ffc95-9hzrh"
Sep 08 13:28:21.715348 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:21.715157    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/511f6abd-9084-449c-a2fb-4c4035463892-tls-certs\") pod \"model-serving-api-7b954ffc95-9hzrh\" (UID: \"511f6abd-9084-449c-a2fb-4c4035463892\") " pod="kserve/model-serving-api-7b954ffc95-9hzrh"
Sep 08 13:28:21.717827 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:21.717800    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/511f6abd-9084-449c-a2fb-4c4035463892-tls-certs\") pod \"model-serving-api-7b954ffc95-9hzrh\" (UID: \"511f6abd-9084-449c-a2fb-4c4035463892\") " pod="kserve/model-serving-api-7b954ffc95-9hzrh"
Sep 08 13:28:21.735315 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:21.735253    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqv9p\" (UniqueName: \"kubernetes.io/projected/511f6abd-9084-449c-a2fb-4c4035463892-kube-api-access-qqv9p\") pod \"model-serving-api-7b954ffc95-9hzrh\" (UID: \"511f6abd-9084-449c-a2fb-4c4035463892\") " pod="kserve/model-serving-api-7b954ffc95-9hzrh"
Sep 08 13:28:21.897441 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:21.897352    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-9hzrh"
Sep 08 13:28:22.181512 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:22.181481    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-9hzrh"]
Sep 08 13:28:22.182919 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:28:22.182895    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod511f6abd_9084_449c_a2fb_4c4035463892.slice/crio-1bf18fa8d96b7daac9887c901fb635cb631a123781e6432f24fff958f4dc040d WatchSource:0}: Error finding container 1bf18fa8d96b7daac9887c901fb635cb631a123781e6432f24fff958f4dc040d: Status 404 returned error can't find the container with id 1bf18fa8d96b7daac9887c901fb635cb631a123781e6432f24fff958f4dc040d
Sep 08 13:28:22.237471 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:22.237441    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-9hzrh" event={"ID":"511f6abd-9084-449c-a2fb-4c4035463892","Type":"ContainerStarted","Data":"1bf18fa8d96b7daac9887c901fb635cb631a123781e6432f24fff958f4dc040d"}
Sep 08 13:28:24.251490 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:24.251442    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-9hzrh" event={"ID":"511f6abd-9084-449c-a2fb-4c4035463892","Type":"ContainerStarted","Data":"ac0c6c402870cee49684c9155ef51003a451f208fea51e34f8632083b2e7fed9"}
Sep 08 13:28:24.251905 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:24.251511    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-9hzrh"
Sep 08 13:28:24.280165 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:24.280108    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-9hzrh" podStartSLOduration=1.595478723 podStartE2EDuration="3.280091663s" totalImagesPullingTime="1.68461294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:28:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:28:24.276811339 +0000 UTC m=+735.737456150" watchObservedRunningTime="2026-09-08 13:28:24.280091663 +0000 UTC m=+735.740736473"
Sep 08 13:28:35.263339 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:28:35.263303    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-9hzrh"
Sep 08 13:30:25.301369 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:25.301317    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz"]
Sep 08 13:30:25.304756 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:25.304733    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz"
Sep 08 13:30:25.308872 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:25.308848    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 08 13:30:25.309902 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:25.309869    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-8v7xm\""
Sep 08 13:30:25.310043 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:25.309954    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 08 13:30:25.310043 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:25.310033    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 08 13:30:25.310167 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:25.310117    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-kq4l4\""
Sep 08 13:30:25.316701 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:25.316659    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz"]
Sep 08 13:30:25.408082 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:25.408044    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phl8g\" (UniqueName: \"kubernetes.io/projected/d5061d8b-8cf8-4bed-b4ae-640ec052ea70-kube-api-access-phl8g\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz\" (UID: \"d5061d8b-8cf8-4bed-b4ae-640ec052ea70\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz"
Sep 08 13:30:25.408263 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:25.408148    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5061d8b-8cf8-4bed-b4ae-640ec052ea70-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz\" (UID: \"d5061d8b-8cf8-4bed-b4ae-640ec052ea70\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz"
Sep 08 13:30:25.509568 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:25.509531    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phl8g\" (UniqueName: \"kubernetes.io/projected/d5061d8b-8cf8-4bed-b4ae-640ec052ea70-kube-api-access-phl8g\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz\" (UID: \"d5061d8b-8cf8-4bed-b4ae-640ec052ea70\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz"
Sep 08 13:30:25.509750 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:25.509588    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5061d8b-8cf8-4bed-b4ae-640ec052ea70-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz\" (UID: \"d5061d8b-8cf8-4bed-b4ae-640ec052ea70\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz"
Sep 08 13:30:25.512721 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:25.512683    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5061d8b-8cf8-4bed-b4ae-640ec052ea70-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz\" (UID: \"d5061d8b-8cf8-4bed-b4ae-640ec052ea70\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz"
Sep 08 13:30:25.518810 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:25.518779    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phl8g\" (UniqueName: \"kubernetes.io/projected/d5061d8b-8cf8-4bed-b4ae-640ec052ea70-kube-api-access-phl8g\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz\" (UID: \"d5061d8b-8cf8-4bed-b4ae-640ec052ea70\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz"
Sep 08 13:30:25.618402 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:25.618274    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz"
Sep 08 13:30:25.901157 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:25.901126    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz"]
Sep 08 13:30:25.902984 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:30:25.902954    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5061d8b_8cf8_4bed_b4ae_640ec052ea70.slice/crio-a2c6cb6f4c3c5b11064fba1fc763e442f4d5faffef20ba5624a48ebaf3ad9472 WatchSource:0}: Error finding container a2c6cb6f4c3c5b11064fba1fc763e442f4d5faffef20ba5624a48ebaf3ad9472: Status 404 returned error can't find the container with id a2c6cb6f4c3c5b11064fba1fc763e442f4d5faffef20ba5624a48ebaf3ad9472
Sep 08 13:30:26.905873 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:26.905839    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz" event={"ID":"d5061d8b-8cf8-4bed-b4ae-640ec052ea70","Type":"ContainerStarted","Data":"a2c6cb6f4c3c5b11064fba1fc763e442f4d5faffef20ba5624a48ebaf3ad9472"}
Sep 08 13:30:27.568618 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:27.568569    2561 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": read tcp 10.134.0.2:40788->10.134.0.31:15021: read: connection reset by peer" start-of-body=
Sep 08 13:30:27.568796 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:27.568634    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn" podUID="db8dab86-bf9e-4760-8002-d3a068f8d45d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": read tcp 10.134.0.2:40788->10.134.0.31:15021: read: connection reset by peer"
Sep 08 13:30:27.912908 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:27.912802    2561 generic.go:356] "Generic (PLEG): container finished" podID="db8dab86-bf9e-4760-8002-d3a068f8d45d" containerID="0f0f9ce4ea1aba582a7f5c141712c08746097b6feb1e6b863062d06922781fef" exitCode=0
Sep 08 13:30:27.912908 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:27.912883    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn" event={"ID":"db8dab86-bf9e-4760-8002-d3a068f8d45d","Type":"ContainerDied","Data":"0f0f9ce4ea1aba582a7f5c141712c08746097b6feb1e6b863062d06922781fef"}
Sep 08 13:30:27.913443 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:27.913398    2561 scope.go:117] "RemoveContainer" containerID="0f0f9ce4ea1aba582a7f5c141712c08746097b6feb1e6b863062d06922781fef"
Sep 08 13:30:27.914421 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:27.914391    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 13:30:27.914525 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:27.914455    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 13:30:27.914525 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:27.914485    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 13:30:28.920842 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:28.920807    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn" event={"ID":"db8dab86-bf9e-4760-8002-d3a068f8d45d","Type":"ContainerStarted","Data":"5e1f42453ab2f9231e676b7e0ddc1f6b2f94bf7d1a1a7170647e3822b74b7d74"}
Sep 08 13:30:29.774719 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:29.774617    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:30:29.774996 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:29.774964    2561 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 08 13:30:29.775070 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:29.775035    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn" podUID="db8dab86-bf9e-4760-8002-d3a068f8d45d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 08 13:30:30.774981 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:30.774937    2561 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 08 13:30:30.775438 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:30.775001    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn" podUID="db8dab86-bf9e-4760-8002-d3a068f8d45d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 08 13:30:31.774981 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:31.774949    2561 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 08 13:30:31.775118 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:31.775026    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn" podUID="db8dab86-bf9e-4760-8002-d3a068f8d45d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 08 13:30:31.938498 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:31.938461    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz" event={"ID":"d5061d8b-8cf8-4bed-b4ae-640ec052ea70","Type":"ContainerStarted","Data":"be6a149601b7b24eac8fb29c3d71c245ba1caa297b539147c415b2e70504ca41"}
Sep 08 13:30:31.938716 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:31.938690    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz"
Sep 08 13:30:32.774186 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:32.774147    2561 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 08 13:30:32.774403 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:32.774208    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn" podUID="db8dab86-bf9e-4760-8002-d3a068f8d45d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 08 13:30:33.774583 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:33.774540    2561 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 08 13:30:33.775015 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:33.774606    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn" podUID="db8dab86-bf9e-4760-8002-d3a068f8d45d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 08 13:30:34.778281 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:34.778248    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:30:34.778809 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:34.778786    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:30:34.779552 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:34.779533    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-b5cvn"
Sep 08 13:30:34.806362 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:30:34.806303    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz" podStartSLOduration=3.981767578 podStartE2EDuration="9.806265673s" totalImagesPullingTime="5.824498095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:30:31.96708165 +0000 UTC m=+863.427726462" watchObservedRunningTime="2026-09-08 13:30:34.806265673 +0000 UTC m=+866.266910485"
Sep 08 13:31:02.949004 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:02.948923    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz"
Sep 08 13:31:08.988121 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:08.988091    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 13:31:08.988684 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:08.988402    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 13:31:09.657761 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:09.657729    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz"]
Sep 08 13:31:12.948663 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:12.948623    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz" podUID="d5061d8b-8cf8-4bed-b4ae-640ec052ea70" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:31:22.949048 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:22.949005    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz" podUID="d5061d8b-8cf8-4bed-b4ae-640ec052ea70" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:31:24.678884 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:24.678839    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz" podUID="d5061d8b-8cf8-4bed-b4ae-640ec052ea70" containerName="main" containerID="cri-o://be6a149601b7b24eac8fb29c3d71c245ba1caa297b539147c415b2e70504ca41" gracePeriod=45
Sep 08 13:31:32.948210 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:32.948172    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz" podUID="d5061d8b-8cf8-4bed-b4ae-640ec052ea70" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:31:32.948697 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:32.948311    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz"
Sep 08 13:31:42.948587 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:42.948551    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz" podUID="d5061d8b-8cf8-4bed-b4ae-640ec052ea70" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:31:52.948690 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:52.948650    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz" podUID="d5061d8b-8cf8-4bed-b4ae-640ec052ea70" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:31:54.928442 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:54.928411    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz"
Sep 08 13:31:54.992851 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:54.992764    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5061d8b-8cf8-4bed-b4ae-640ec052ea70-tls-certs\") pod \"d5061d8b-8cf8-4bed-b4ae-640ec052ea70\" (UID: \"d5061d8b-8cf8-4bed-b4ae-640ec052ea70\") "
Sep 08 13:31:54.992851 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:54.992814    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phl8g\" (UniqueName: \"kubernetes.io/projected/d5061d8b-8cf8-4bed-b4ae-640ec052ea70-kube-api-access-phl8g\") pod \"d5061d8b-8cf8-4bed-b4ae-640ec052ea70\" (UID: \"d5061d8b-8cf8-4bed-b4ae-640ec052ea70\") "
Sep 08 13:31:54.995023 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:54.994990    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5061d8b-8cf8-4bed-b4ae-640ec052ea70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5061d8b-8cf8-4bed-b4ae-640ec052ea70" (UID: "d5061d8b-8cf8-4bed-b4ae-640ec052ea70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:31:54.995126 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:54.995034    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5061d8b-8cf8-4bed-b4ae-640ec052ea70-kube-api-access-phl8g" (OuterVolumeSpecName: "kube-api-access-phl8g") pod "d5061d8b-8cf8-4bed-b4ae-640ec052ea70" (UID: "d5061d8b-8cf8-4bed-b4ae-640ec052ea70"). InnerVolumeSpecName "kube-api-access-phl8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:31:55.093890 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:55.093848    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5061d8b-8cf8-4bed-b4ae-640ec052ea70-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:31:55.093890 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:55.093884    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phl8g\" (UniqueName: \"kubernetes.io/projected/d5061d8b-8cf8-4bed-b4ae-640ec052ea70-kube-api-access-phl8g\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:31:55.386178 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:55.386089    2561 generic.go:356] "Generic (PLEG): container finished" podID="d5061d8b-8cf8-4bed-b4ae-640ec052ea70" containerID="be6a149601b7b24eac8fb29c3d71c245ba1caa297b539147c415b2e70504ca41" exitCode=0
Sep 08 13:31:55.386178 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:55.386120    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz" event={"ID":"d5061d8b-8cf8-4bed-b4ae-640ec052ea70","Type":"ContainerDied","Data":"be6a149601b7b24eac8fb29c3d71c245ba1caa297b539147c415b2e70504ca41"}
Sep 08 13:31:55.386495 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:55.386182    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz" event={"ID":"d5061d8b-8cf8-4bed-b4ae-640ec052ea70","Type":"ContainerDied","Data":"a2c6cb6f4c3c5b11064fba1fc763e442f4d5faffef20ba5624a48ebaf3ad9472"}
Sep 08 13:31:55.386495 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:55.386205    2561 scope.go:117] "RemoveContainer" containerID="be6a149601b7b24eac8fb29c3d71c245ba1caa297b539147c415b2e70504ca41"
Sep 08 13:31:55.386495 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:55.386131    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz"
Sep 08 13:31:55.395239 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:55.395217    2561 scope.go:117] "RemoveContainer" containerID="be6a149601b7b24eac8fb29c3d71c245ba1caa297b539147c415b2e70504ca41"
Sep 08 13:31:55.395538 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:31:55.395520    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6a149601b7b24eac8fb29c3d71c245ba1caa297b539147c415b2e70504ca41\": container with ID starting with be6a149601b7b24eac8fb29c3d71c245ba1caa297b539147c415b2e70504ca41 not found: ID does not exist" containerID="be6a149601b7b24eac8fb29c3d71c245ba1caa297b539147c415b2e70504ca41"
Sep 08 13:31:55.395603 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:55.395546    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6a149601b7b24eac8fb29c3d71c245ba1caa297b539147c415b2e70504ca41"} err="failed to get container status \"be6a149601b7b24eac8fb29c3d71c245ba1caa297b539147c415b2e70504ca41\": rpc error: code = NotFound desc = could not find container \"be6a149601b7b24eac8fb29c3d71c245ba1caa297b539147c415b2e70504ca41\": container with ID starting with be6a149601b7b24eac8fb29c3d71c245ba1caa297b539147c415b2e70504ca41 not found: ID does not exist"
Sep 08 13:31:55.402526 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:55.402499    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz"]
Sep 08 13:31:55.407110 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:55.407089    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-spvcz"]
Sep 08 13:31:57.007244 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:31:57.007211    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5061d8b-8cf8-4bed-b4ae-640ec052ea70" path="/var/lib/kubelet/pods/d5061d8b-8cf8-4bed-b4ae-640ec052ea70/volumes"
Sep 08 13:32:20.570102 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:20.570061    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc"]
Sep 08 13:32:20.570766 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:20.570581    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5061d8b-8cf8-4bed-b4ae-640ec052ea70" containerName="main"
Sep 08 13:32:20.570766 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:20.570602    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5061d8b-8cf8-4bed-b4ae-640ec052ea70" containerName="main"
Sep 08 13:32:20.570766 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:20.570719    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5061d8b-8cf8-4bed-b4ae-640ec052ea70" containerName="main"
Sep 08 13:32:20.574931 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:20.574908    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc"
Sep 08 13:32:20.578147 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:20.578112    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-4n92l\""
Sep 08 13:32:20.578147 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:20.578142    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 08 13:32:20.578353 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:20.578119    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 08 13:32:20.578353 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:20.578203    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-dst2n\""
Sep 08 13:32:20.578474 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:20.578395    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 08 13:32:20.587457 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:20.587435    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc"]
Sep 08 13:32:20.726009 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:20.725972    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp472\" (UniqueName: \"kubernetes.io/projected/0b39adf7-863d-430b-8174-7440c799871e-kube-api-access-zp472\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc\" (UID: \"0b39adf7-863d-430b-8174-7440c799871e\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc"
Sep 08 13:32:20.726196 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:20.726090    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b39adf7-863d-430b-8174-7440c799871e-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc\" (UID: \"0b39adf7-863d-430b-8174-7440c799871e\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc"
Sep 08 13:32:20.827532 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:20.827429    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b39adf7-863d-430b-8174-7440c799871e-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc\" (UID: \"0b39adf7-863d-430b-8174-7440c799871e\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc"
Sep 08 13:32:20.827532 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:20.827494    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp472\" (UniqueName: \"kubernetes.io/projected/0b39adf7-863d-430b-8174-7440c799871e-kube-api-access-zp472\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc\" (UID: \"0b39adf7-863d-430b-8174-7440c799871e\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc"
Sep 08 13:32:20.830161 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:20.830124    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b39adf7-863d-430b-8174-7440c799871e-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc\" (UID: \"0b39adf7-863d-430b-8174-7440c799871e\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc"
Sep 08 13:32:20.836640 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:20.836609    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp472\" (UniqueName: \"kubernetes.io/projected/0b39adf7-863d-430b-8174-7440c799871e-kube-api-access-zp472\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc\" (UID: \"0b39adf7-863d-430b-8174-7440c799871e\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc"
Sep 08 13:32:20.888689 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:20.888650    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc"
Sep 08 13:32:21.167009 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:21.166982    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc"]
Sep 08 13:32:21.169454 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:32:21.169426    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b39adf7_863d_430b_8174_7440c799871e.slice/crio-6cbf25aeadbcd97afd9892ad24b7dfe0f60045e2c1b10368159574e59fc125a2 WatchSource:0}: Error finding container 6cbf25aeadbcd97afd9892ad24b7dfe0f60045e2c1b10368159574e59fc125a2: Status 404 returned error can't find the container with id 6cbf25aeadbcd97afd9892ad24b7dfe0f60045e2c1b10368159574e59fc125a2
Sep 08 13:32:21.527086 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:21.527049    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc" event={"ID":"0b39adf7-863d-430b-8174-7440c799871e","Type":"ContainerStarted","Data":"1aa675a93866a397d593a957caa20d9a3afb09c17e4401cc444acdf997b736b5"}
Sep 08 13:32:21.527275 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:21.527106    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc" event={"ID":"0b39adf7-863d-430b-8174-7440c799871e","Type":"ContainerStarted","Data":"6cbf25aeadbcd97afd9892ad24b7dfe0f60045e2c1b10368159574e59fc125a2"}
Sep 08 13:32:21.527275 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:21.527166    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc"
Sep 08 13:32:21.547587 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:21.547530    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc" podStartSLOduration=1.547512472 podStartE2EDuration="1.547512472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:32:21.546040814 +0000 UTC m=+973.006685626" watchObservedRunningTime="2026-09-08 13:32:21.547512472 +0000 UTC m=+973.008157308"
Sep 08 13:32:52.535590 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:32:52.535557    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc"
Sep 08 13:33:00.045994 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:00.045952    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc"]
Sep 08 13:33:02.535824 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:02.535782    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc" podUID="0b39adf7-863d-430b-8174-7440c799871e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:33:12.535586 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:12.535544    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc" podUID="0b39adf7-863d-430b-8174-7440c799871e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:33:15.068159 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:15.068069    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc" podUID="0b39adf7-863d-430b-8174-7440c799871e" containerName="main" containerID="cri-o://1aa675a93866a397d593a957caa20d9a3afb09c17e4401cc444acdf997b736b5" gracePeriod=45
Sep 08 13:33:22.535657 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:22.535614    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc" podUID="0b39adf7-863d-430b-8174-7440c799871e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:33:22.536072 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:22.535760    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc"
Sep 08 13:33:32.535187 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:32.535144    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc" podUID="0b39adf7-863d-430b-8174-7440c799871e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:33:42.535587 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:42.535547    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc" podUID="0b39adf7-863d-430b-8174-7440c799871e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:33:45.331930 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:45.331868    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc"
Sep 08 13:33:45.375967 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:45.375932    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b39adf7-863d-430b-8174-7440c799871e-tls-certs\") pod \"0b39adf7-863d-430b-8174-7440c799871e\" (UID: \"0b39adf7-863d-430b-8174-7440c799871e\") "
Sep 08 13:33:45.376159 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:45.376000    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zp472\" (UniqueName: \"kubernetes.io/projected/0b39adf7-863d-430b-8174-7440c799871e-kube-api-access-zp472\") pod \"0b39adf7-863d-430b-8174-7440c799871e\" (UID: \"0b39adf7-863d-430b-8174-7440c799871e\") "
Sep 08 13:33:45.378229 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:45.378200    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b39adf7-863d-430b-8174-7440c799871e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b39adf7-863d-430b-8174-7440c799871e" (UID: "0b39adf7-863d-430b-8174-7440c799871e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:33:45.378370 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:45.378251    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b39adf7-863d-430b-8174-7440c799871e-kube-api-access-zp472" (OuterVolumeSpecName: "kube-api-access-zp472") pod "0b39adf7-863d-430b-8174-7440c799871e" (UID: "0b39adf7-863d-430b-8174-7440c799871e"). InnerVolumeSpecName "kube-api-access-zp472". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:33:45.477800 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:45.477763    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b39adf7-863d-430b-8174-7440c799871e-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:33:45.477800 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:45.477795    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp472\" (UniqueName: \"kubernetes.io/projected/0b39adf7-863d-430b-8174-7440c799871e-kube-api-access-zp472\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:33:45.984727 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:45.984689    2561 generic.go:356] "Generic (PLEG): container finished" podID="0b39adf7-863d-430b-8174-7440c799871e" containerID="1aa675a93866a397d593a957caa20d9a3afb09c17e4401cc444acdf997b736b5" exitCode=0
Sep 08 13:33:45.984900 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:45.984741    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc"
Sep 08 13:33:45.984900 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:45.984781    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc" event={"ID":"0b39adf7-863d-430b-8174-7440c799871e","Type":"ContainerDied","Data":"1aa675a93866a397d593a957caa20d9a3afb09c17e4401cc444acdf997b736b5"}
Sep 08 13:33:45.984900 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:45.984841    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc" event={"ID":"0b39adf7-863d-430b-8174-7440c799871e","Type":"ContainerDied","Data":"6cbf25aeadbcd97afd9892ad24b7dfe0f60045e2c1b10368159574e59fc125a2"}
Sep 08 13:33:45.984900 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:45.984864    2561 scope.go:117] "RemoveContainer" containerID="1aa675a93866a397d593a957caa20d9a3afb09c17e4401cc444acdf997b736b5"
Sep 08 13:33:45.994141 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:45.994123    2561 scope.go:117] "RemoveContainer" containerID="1aa675a93866a397d593a957caa20d9a3afb09c17e4401cc444acdf997b736b5"
Sep 08 13:33:45.994496 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:33:45.994473    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa675a93866a397d593a957caa20d9a3afb09c17e4401cc444acdf997b736b5\": container with ID starting with 1aa675a93866a397d593a957caa20d9a3afb09c17e4401cc444acdf997b736b5 not found: ID does not exist" containerID="1aa675a93866a397d593a957caa20d9a3afb09c17e4401cc444acdf997b736b5"
Sep 08 13:33:45.994546 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:45.994506    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa675a93866a397d593a957caa20d9a3afb09c17e4401cc444acdf997b736b5"} err="failed to get container status \"1aa675a93866a397d593a957caa20d9a3afb09c17e4401cc444acdf997b736b5\": rpc error: code = NotFound desc = could not find container \"1aa675a93866a397d593a957caa20d9a3afb09c17e4401cc444acdf997b736b5\": container with ID starting with 1aa675a93866a397d593a957caa20d9a3afb09c17e4401cc444acdf997b736b5 not found: ID does not exist"
Sep 08 13:33:46.006155 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:46.006126    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc"]
Sep 08 13:33:46.009282 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:46.009245    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-qwvlc"]
Sep 08 13:33:47.006184 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:47.006153    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b39adf7-863d-430b-8174-7440c799871e" path="/var/lib/kubelet/pods/0b39adf7-863d-430b-8174-7440c799871e/volumes"
Sep 08 13:33:50.374827 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.374729    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"]
Sep 08 13:33:50.375210 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.375118    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b39adf7-863d-430b-8174-7440c799871e" containerName="main"
Sep 08 13:33:50.375210 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.375130    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b39adf7-863d-430b-8174-7440c799871e" containerName="main"
Sep 08 13:33:50.375210 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.375190    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b39adf7-863d-430b-8174-7440c799871e" containerName="main"
Sep 08 13:33:50.379581 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.379556    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.382634 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.382606    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 08 13:33:50.382794 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.382722    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-bz9hn\""
Sep 08 13:33:50.383050 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.383030    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 08 13:33:50.383178 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.383156    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 08 13:33:50.396824 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.396791    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"]
Sep 08 13:33:50.526362 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.526322    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.526362 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.526365    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds5zw\" (UniqueName: \"kubernetes.io/projected/5e266ed1-1aaa-4ea4-a011-8091d01f6504-kube-api-access-ds5zw\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.526635 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.526424    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.526635 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.526491    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.526635 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.526511    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.526635 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.526541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.526635 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.526568    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.526635 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.526608    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.526635 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.526629    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.627576 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.627480    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.627576 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.627534    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.627806 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.627586    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.627806 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.627612    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.627806 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.627647    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.627806 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.627668    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds5zw\" (UniqueName: \"kubernetes.io/projected/5e266ed1-1aaa-4ea4-a011-8091d01f6504-kube-api-access-ds5zw\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.627806 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.627715    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.627806 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.627769    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.627806 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.627799    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.628134 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.627995    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.628134 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.628019    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.628134 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.628026    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.628233 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.628187    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.628490 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.628466    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.630157 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.630126    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.630339 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.630314    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.636825 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.636800    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds5zw\" (UniqueName: \"kubernetes.io/projected/5e266ed1-1aaa-4ea4-a011-8091d01f6504-kube-api-access-ds5zw\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.637060 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.637037    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-ccghd\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.690359 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.690272    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:50.973735 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:33:50.973704    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e266ed1_1aaa_4ea4_a011_8091d01f6504.slice/crio-92dd6a28fe6bc7734aeddeda1987e9c3c6d0bc484fbdc71ae3199a1d00dca35b WatchSource:0}: Error finding container 92dd6a28fe6bc7734aeddeda1987e9c3c6d0bc484fbdc71ae3199a1d00dca35b: Status 404 returned error can't find the container with id 92dd6a28fe6bc7734aeddeda1987e9c3c6d0bc484fbdc71ae3199a1d00dca35b
Sep 08 13:33:50.975946 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.975920    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"]
Sep 08 13:33:50.976351 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.976280    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 13:33:50.976424 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.976395    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 13:33:50.976497 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:50.976427    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 13:33:51.013966 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:51.013930    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd" event={"ID":"5e266ed1-1aaa-4ea4-a011-8091d01f6504","Type":"ContainerStarted","Data":"92dd6a28fe6bc7734aeddeda1987e9c3c6d0bc484fbdc71ae3199a1d00dca35b"}
Sep 08 13:33:52.020647 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:52.020611    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd" event={"ID":"5e266ed1-1aaa-4ea4-a011-8091d01f6504","Type":"ContainerStarted","Data":"087b471785534469835cd6f2a68d440a5f6d9d9920d30be9cc54e7e8ab0450ea"}
Sep 08 13:33:52.043364 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:52.043272    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd" podStartSLOduration=2.043250493 podStartE2EDuration="2.043250493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:33:52.040683419 +0000 UTC m=+1063.501328231" watchObservedRunningTime="2026-09-08 13:33:52.043250493 +0000 UTC m=+1063.503895307"
Sep 08 13:33:52.690525 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:52.690478    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:52.695522 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:52.695492    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:53.026421 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:53.026326    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:33:53.027546 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:33:53.027529    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:34:14.408617 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.408584    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"]
Sep 08 13:34:14.413694 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.413671    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:14.416545 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.416524    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-n6kwb\""
Sep 08 13:34:14.416639 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.416523    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 08 13:34:14.421964 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.421938    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"]
Sep 08 13:34:14.441601 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.441572    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:14.441758 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.441710    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:14.441758 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.441743    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:14.441874 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.441762    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65c770f3-41b0-4863-8052-fb44b468fd77-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:14.441874 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.441788    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:14.441952 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.441868    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f6zc\" (UniqueName: \"kubernetes.io/projected/65c770f3-41b0-4863-8052-fb44b468fd77-kube-api-access-9f6zc\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:14.543363 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.543326    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f6zc\" (UniqueName: \"kubernetes.io/projected/65c770f3-41b0-4863-8052-fb44b468fd77-kube-api-access-9f6zc\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:14.543540 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.543379    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:14.543540 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.543465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:14.543540 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.543498    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:14.543540 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.543520    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65c770f3-41b0-4863-8052-fb44b468fd77-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:14.543752 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.543556    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:14.543909 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.543881    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:14.543986 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.543923    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:14.544027 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.543988    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:14.546124 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.546093    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:14.546270 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.546251    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65c770f3-41b0-4863-8052-fb44b468fd77-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:14.552272 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.552250    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f6zc\" (UniqueName: \"kubernetes.io/projected/65c770f3-41b0-4863-8052-fb44b468fd77-kube-api-access-9f6zc\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:14.725634 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:14.725596    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:15.007742 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:15.007709    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"]
Sep 08 13:34:15.008668 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:34:15.008637    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c770f3_41b0_4863_8052_fb44b468fd77.slice/crio-155415655c54da2cd6d63463514908258e0b212f5d87165e1cd3f26e42af9248 WatchSource:0}: Error finding container 155415655c54da2cd6d63463514908258e0b212f5d87165e1cd3f26e42af9248: Status 404 returned error can't find the container with id 155415655c54da2cd6d63463514908258e0b212f5d87165e1cd3f26e42af9248
Sep 08 13:34:15.010425 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:15.010407    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:34:15.150847 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:15.150815    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m" event={"ID":"65c770f3-41b0-4863-8052-fb44b468fd77","Type":"ContainerStarted","Data":"155415655c54da2cd6d63463514908258e0b212f5d87165e1cd3f26e42af9248"}
Sep 08 13:34:17.170610 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.170572    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m" event={"ID":"65c770f3-41b0-4863-8052-fb44b468fd77","Type":"ContainerStarted","Data":"87e3b5528eb680eb57fdfcb799500be13b0923c91e955958a9d382459cfe6832"}
Sep 08 13:34:17.193012 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.192959    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m" podStartSLOduration=1.217950869 podStartE2EDuration="3.192943646s" totalImagesPullingTime="1.974992777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:34:17.18989087 +0000 UTC m=+1088.650535679" watchObservedRunningTime="2026-09-08 13:34:17.192943646 +0000 UTC m=+1088.653588457"
Sep 08 13:34:17.522622 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.522591    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"]
Sep 08 13:34:17.526506 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.526476    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.529024 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.528995    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 08 13:34:17.529165 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.529019    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 08 13:34:17.529165 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.529006    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 08 13:34:17.529165 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.529128    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-x98dx\""
Sep 08 13:34:17.557362 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.541003    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"]
Sep 08 13:34:17.582181 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.582145    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.582399 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.582215    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d9640d-8980-41f5-a657-4286e85ee4fb-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.582399 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.582320    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.582399 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.582368    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.582399 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.582398    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.582590 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.582417    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.582590 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.582461    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.582590 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.582490    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.582590 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.582537    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bhbj\" (UniqueName: \"kubernetes.io/projected/40d9640d-8980-41f5-a657-4286e85ee4fb-kube-api-access-6bhbj\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.683954 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.683915    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d9640d-8980-41f5-a657-4286e85ee4fb-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.684147 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.683959    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.684147 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.683991    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.684147 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.684030    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.684147 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.684056    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.684147 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.684093    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.684147 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.684117    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.684147 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.684144    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bhbj\" (UniqueName: \"kubernetes.io/projected/40d9640d-8980-41f5-a657-4286e85ee4fb-kube-api-access-6bhbj\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.684544 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.684229    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.684610 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.684589    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.684669 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.684622    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.684810 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.684789    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d9640d-8980-41f5-a657-4286e85ee4fb-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.684927 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.684903    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.684984 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.684941    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.686608 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.686579    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.686802 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.686782    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.692517 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.692494    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.692624 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.692586    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bhbj\" (UniqueName: \"kubernetes.io/projected/40d9640d-8980-41f5-a657-4286e85ee4fb-kube-api-access-6bhbj\") pod \"router-gateway-1-openshift-default-66bbccc5fb-t5mt6\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:17.726427 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.726380    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:17.837492 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:17.837381    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:18.136156 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:18.136123    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"]
Sep 08 13:34:18.137535 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:34:18.137512    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d9640d_8980_41f5_a657_4286e85ee4fb.slice/crio-dde83503209cf643893a1bd542157f9ad772e6c7cecd7e5bb7f4921c471a2c22 WatchSource:0}: Error finding container dde83503209cf643893a1bd542157f9ad772e6c7cecd7e5bb7f4921c471a2c22: Status 404 returned error can't find the container with id dde83503209cf643893a1bd542157f9ad772e6c7cecd7e5bb7f4921c471a2c22
Sep 08 13:34:18.139688 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:18.139640    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 13:34:18.139764 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:18.139725    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 13:34:18.139764 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:18.139753    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 13:34:18.178741 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:18.178711    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6" event={"ID":"40d9640d-8980-41f5-a657-4286e85ee4fb","Type":"ContainerStarted","Data":"dde83503209cf643893a1bd542157f9ad772e6c7cecd7e5bb7f4921c471a2c22"}
Sep 08 13:34:19.186136 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:19.186082    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6" event={"ID":"40d9640d-8980-41f5-a657-4286e85ee4fb","Type":"ContainerStarted","Data":"1e70bb1cb24636cc1729fa8edf95f40a584d91822352b9c4ed628be3f855fdb9"}
Sep 08 13:34:19.207191 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:19.207138    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6" podStartSLOduration=2.207121945 podStartE2EDuration="2.207121945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:34:19.206298658 +0000 UTC m=+1090.666943463" watchObservedRunningTime="2026-09-08 13:34:19.207121945 +0000 UTC m=+1090.667766753"
Sep 08 13:34:19.838029 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:19.837985    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:19.848752 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:19.848713    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:20.192346 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:20.192318    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:20.193391 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:20.193374    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:22.221568 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.221531    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"]
Sep 08 13:34:22.229049 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.229013    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"]
Sep 08 13:34:22.229312 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.229252    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd" podUID="5e266ed1-1aaa-4ea4-a011-8091d01f6504" containerName="istio-proxy" containerID="cri-o://087b471785534469835cd6f2a68d440a5f6d9d9920d30be9cc54e7e8ab0450ea" gracePeriod=30
Sep 08 13:34:22.233119 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:34:22.233085    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:34:22.233119 ip-10-0-130-178 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:34:22.233119 ip-10-0-130-178 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 08 13:34:22.233119 ip-10-0-130-178 kubenswrapper[2561]:  > containerID="87e3b5528eb680eb57fdfcb799500be13b0923c91e955958a9d382459cfe6832" cmd=["/bin/sleep","15"]
Sep 08 13:34:22.233382 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.233140    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m" message="" err=<
Sep 08 13:34:22.233382 ip-10-0-130-178 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:34:22.233382 ip-10-0-130-178 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 08 13:34:22.233382 ip-10-0-130-178 kubenswrapper[2561]:  >
Sep 08 13:34:22.233382 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:34:22.233153    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:34:22.233382 ip-10-0-130-178 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:34:22.233382 ip-10-0-130-178 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 08 13:34:22.233382 ip-10-0-130-178 kubenswrapper[2561]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m" podUID="65c770f3-41b0-4863-8052-fb44b468fd77" containerName="main" containerID="cri-o://87e3b5528eb680eb57fdfcb799500be13b0923c91e955958a9d382459cfe6832"
Sep 08 13:34:22.233382 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.233193    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m" podUID="65c770f3-41b0-4863-8052-fb44b468fd77" containerName="main" containerID="cri-o://87e3b5528eb680eb57fdfcb799500be13b0923c91e955958a9d382459cfe6832" gracePeriod=60
Sep 08 13:34:22.506192 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.506166    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:22.527530 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.527497    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-model-cache\") pod \"65c770f3-41b0-4863-8052-fb44b468fd77\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") "
Sep 08 13:34:22.527699 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.527559    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-home\") pod \"65c770f3-41b0-4863-8052-fb44b468fd77\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") "
Sep 08 13:34:22.527699 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.527591    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-tmp-dir\") pod \"65c770f3-41b0-4863-8052-fb44b468fd77\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") "
Sep 08 13:34:22.527699 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.527635    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65c770f3-41b0-4863-8052-fb44b468fd77-tls-certs\") pod \"65c770f3-41b0-4863-8052-fb44b468fd77\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") "
Sep 08 13:34:22.527699 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.527696    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f6zc\" (UniqueName: \"kubernetes.io/projected/65c770f3-41b0-4863-8052-fb44b468fd77-kube-api-access-9f6zc\") pod \"65c770f3-41b0-4863-8052-fb44b468fd77\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") "
Sep 08 13:34:22.527851 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.527734    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-dshm\") pod \"65c770f3-41b0-4863-8052-fb44b468fd77\" (UID: \"65c770f3-41b0-4863-8052-fb44b468fd77\") "
Sep 08 13:34:22.527851 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.527793    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-model-cache" (OuterVolumeSpecName: "model-cache") pod "65c770f3-41b0-4863-8052-fb44b468fd77" (UID: "65c770f3-41b0-4863-8052-fb44b468fd77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:34:22.528011 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.527988    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "65c770f3-41b0-4863-8052-fb44b468fd77" (UID: "65c770f3-41b0-4863-8052-fb44b468fd77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:34:22.528091 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.528019    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-model-cache\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:22.528206 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.528173    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-home" (OuterVolumeSpecName: "home") pod "65c770f3-41b0-4863-8052-fb44b468fd77" (UID: "65c770f3-41b0-4863-8052-fb44b468fd77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:34:22.530901 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.530631    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65c770f3-41b0-4863-8052-fb44b468fd77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65c770f3-41b0-4863-8052-fb44b468fd77" (UID: "65c770f3-41b0-4863-8052-fb44b468fd77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:34:22.531034 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.530913    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-dshm" (OuterVolumeSpecName: "dshm") pod "65c770f3-41b0-4863-8052-fb44b468fd77" (UID: "65c770f3-41b0-4863-8052-fb44b468fd77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:34:22.531411 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.531351    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c770f3-41b0-4863-8052-fb44b468fd77-kube-api-access-9f6zc" (OuterVolumeSpecName: "kube-api-access-9f6zc") pod "65c770f3-41b0-4863-8052-fb44b468fd77" (UID: "65c770f3-41b0-4863-8052-fb44b468fd77"). InnerVolumeSpecName "kube-api-access-9f6zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:34:22.629237 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.629198    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-home\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:22.629237 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.629231    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-tmp-dir\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:22.629237 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.629241    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65c770f3-41b0-4863-8052-fb44b468fd77-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:22.629498 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.629250    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f6zc\" (UniqueName: \"kubernetes.io/projected/65c770f3-41b0-4863-8052-fb44b468fd77-kube-api-access-9f6zc\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:22.629498 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:22.629260    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65c770f3-41b0-4863-8052-fb44b468fd77-dshm\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:23.211986 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:23.211952    2561 generic.go:356] "Generic (PLEG): container finished" podID="65c770f3-41b0-4863-8052-fb44b468fd77" containerID="87e3b5528eb680eb57fdfcb799500be13b0923c91e955958a9d382459cfe6832" exitCode=0
Sep 08 13:34:23.212177 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:23.211999    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"
Sep 08 13:34:23.212177 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:23.212043    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m" event={"ID":"65c770f3-41b0-4863-8052-fb44b468fd77","Type":"ContainerDied","Data":"87e3b5528eb680eb57fdfcb799500be13b0923c91e955958a9d382459cfe6832"}
Sep 08 13:34:23.212177 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:23.212108    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m" event={"ID":"65c770f3-41b0-4863-8052-fb44b468fd77","Type":"ContainerDied","Data":"155415655c54da2cd6d63463514908258e0b212f5d87165e1cd3f26e42af9248"}
Sep 08 13:34:23.212177 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:23.212155    2561 scope.go:117] "RemoveContainer" containerID="87e3b5528eb680eb57fdfcb799500be13b0923c91e955958a9d382459cfe6832"
Sep 08 13:34:23.222970 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:23.222946    2561 scope.go:117] "RemoveContainer" containerID="87e3b5528eb680eb57fdfcb799500be13b0923c91e955958a9d382459cfe6832"
Sep 08 13:34:23.223302 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:34:23.223231    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87e3b5528eb680eb57fdfcb799500be13b0923c91e955958a9d382459cfe6832\": container with ID starting with 87e3b5528eb680eb57fdfcb799500be13b0923c91e955958a9d382459cfe6832 not found: ID does not exist" containerID="87e3b5528eb680eb57fdfcb799500be13b0923c91e955958a9d382459cfe6832"
Sep 08 13:34:23.223302 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:23.223255    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87e3b5528eb680eb57fdfcb799500be13b0923c91e955958a9d382459cfe6832"} err="failed to get container status \"87e3b5528eb680eb57fdfcb799500be13b0923c91e955958a9d382459cfe6832\": rpc error: code = NotFound desc = could not find container \"87e3b5528eb680eb57fdfcb799500be13b0923c91e955958a9d382459cfe6832\": container with ID starting with 87e3b5528eb680eb57fdfcb799500be13b0923c91e955958a9d382459cfe6832 not found: ID does not exist"
Sep 08 13:34:23.230040 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:23.230014    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"]
Sep 08 13:34:23.233485 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:23.233462    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xqt6m"]
Sep 08 13:34:24.067473 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.067429    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd" podUID="5e266ed1-1aaa-4ea4-a011-8091d01f6504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 08 13:34:24.483069 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.483037    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:34:24.547829 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.547793    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-envoy\") pod \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") "
Sep 08 13:34:24.548013 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.547856    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ds5zw\" (UniqueName: \"kubernetes.io/projected/5e266ed1-1aaa-4ea4-a011-8091d01f6504-kube-api-access-ds5zw\") pod \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") "
Sep 08 13:34:24.548013 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.547916    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istiod-ca-cert\") pod \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") "
Sep 08 13:34:24.548013 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.547946    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-credential-socket\") pod \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") "
Sep 08 13:34:24.548013 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.547983    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-workload-certs\") pod \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") "
Sep 08 13:34:24.548013 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.548008    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-token\") pod \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") "
Sep 08 13:34:24.548322 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.548067    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-podinfo\") pod \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") "
Sep 08 13:34:24.548322 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.548123    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-workload-socket\") pod \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") "
Sep 08 13:34:24.548322 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.548165    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-data\") pod \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\" (UID: \"5e266ed1-1aaa-4ea4-a011-8091d01f6504\") "
Sep 08 13:34:24.548840 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.548744    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-data" (OuterVolumeSpecName: "istio-data") pod "5e266ed1-1aaa-4ea4-a011-8091d01f6504" (UID: "5e266ed1-1aaa-4ea4-a011-8091d01f6504"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:34:24.549003 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.548978    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5e266ed1-1aaa-4ea4-a011-8091d01f6504" (UID: "5e266ed1-1aaa-4ea4-a011-8091d01f6504"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:34:24.549115 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.549086    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5e266ed1-1aaa-4ea4-a011-8091d01f6504" (UID: "5e266ed1-1aaa-4ea4-a011-8091d01f6504"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:34:24.549280 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.549257    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5e266ed1-1aaa-4ea4-a011-8091d01f6504" (UID: "5e266ed1-1aaa-4ea4-a011-8091d01f6504"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:34:24.549424 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.549403    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5e266ed1-1aaa-4ea4-a011-8091d01f6504" (UID: "5e266ed1-1aaa-4ea4-a011-8091d01f6504"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:34:24.552145 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.552096    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e266ed1-1aaa-4ea4-a011-8091d01f6504-kube-api-access-ds5zw" (OuterVolumeSpecName: "kube-api-access-ds5zw") pod "5e266ed1-1aaa-4ea4-a011-8091d01f6504" (UID: "5e266ed1-1aaa-4ea4-a011-8091d01f6504"). InnerVolumeSpecName "kube-api-access-ds5zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:34:24.552145 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.552105    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-token" (OuterVolumeSpecName: "istio-token") pod "5e266ed1-1aaa-4ea4-a011-8091d01f6504" (UID: "5e266ed1-1aaa-4ea4-a011-8091d01f6504"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:34:24.552368 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.552272    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5e266ed1-1aaa-4ea4-a011-8091d01f6504" (UID: "5e266ed1-1aaa-4ea4-a011-8091d01f6504"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:34:24.552635 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.552609    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5e266ed1-1aaa-4ea4-a011-8091d01f6504" (UID: "5e266ed1-1aaa-4ea4-a011-8091d01f6504"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 13:34:24.648986 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.648900    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-envoy\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:24.648986 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.648933    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds5zw\" (UniqueName: \"kubernetes.io/projected/5e266ed1-1aaa-4ea4-a011-8091d01f6504-kube-api-access-ds5zw\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:24.648986 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.648944    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istiod-ca-cert\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:24.648986 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.648954    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-credential-socket\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:24.648986 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.648962    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-workload-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:24.648986 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.648971    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-token\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:24.648986 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.648978    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-podinfo\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:24.648986 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.648985    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-workload-socket\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:24.649377 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:24.649004    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e266ed1-1aaa-4ea4-a011-8091d01f6504-istio-data\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:25.006653 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:25.006623    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65c770f3-41b0-4863-8052-fb44b468fd77" path="/var/lib/kubelet/pods/65c770f3-41b0-4863-8052-fb44b468fd77/volumes"
Sep 08 13:34:25.225887 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:25.225851    2561 generic.go:356] "Generic (PLEG): container finished" podID="5e266ed1-1aaa-4ea4-a011-8091d01f6504" containerID="087b471785534469835cd6f2a68d440a5f6d9d9920d30be9cc54e7e8ab0450ea" exitCode=0
Sep 08 13:34:25.226089 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:25.225897    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"
Sep 08 13:34:25.226089 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:25.225925    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd" event={"ID":"5e266ed1-1aaa-4ea4-a011-8091d01f6504","Type":"ContainerDied","Data":"087b471785534469835cd6f2a68d440a5f6d9d9920d30be9cc54e7e8ab0450ea"}
Sep 08 13:34:25.226089 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:25.225980    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd" event={"ID":"5e266ed1-1aaa-4ea4-a011-8091d01f6504","Type":"ContainerDied","Data":"92dd6a28fe6bc7734aeddeda1987e9c3c6d0bc484fbdc71ae3199a1d00dca35b"}
Sep 08 13:34:25.226089 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:25.226007    2561 scope.go:117] "RemoveContainer" containerID="087b471785534469835cd6f2a68d440a5f6d9d9920d30be9cc54e7e8ab0450ea"
Sep 08 13:34:25.235405 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:25.235381    2561 scope.go:117] "RemoveContainer" containerID="087b471785534469835cd6f2a68d440a5f6d9d9920d30be9cc54e7e8ab0450ea"
Sep 08 13:34:25.235711 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:34:25.235686    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087b471785534469835cd6f2a68d440a5f6d9d9920d30be9cc54e7e8ab0450ea\": container with ID starting with 087b471785534469835cd6f2a68d440a5f6d9d9920d30be9cc54e7e8ab0450ea not found: ID does not exist" containerID="087b471785534469835cd6f2a68d440a5f6d9d9920d30be9cc54e7e8ab0450ea"
Sep 08 13:34:25.235759 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:25.235722    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087b471785534469835cd6f2a68d440a5f6d9d9920d30be9cc54e7e8ab0450ea"} err="failed to get container status \"087b471785534469835cd6f2a68d440a5f6d9d9920d30be9cc54e7e8ab0450ea\": rpc error: code = NotFound desc = could not find container \"087b471785534469835cd6f2a68d440a5f6d9d9920d30be9cc54e7e8ab0450ea\": container with ID starting with 087b471785534469835cd6f2a68d440a5f6d9d9920d30be9cc54e7e8ab0450ea not found: ID does not exist"
Sep 08 13:34:25.244368 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:25.244340    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"]
Sep 08 13:34:25.250536 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:25.250505    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-ccghd"]
Sep 08 13:34:27.007178 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:27.007134    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e266ed1-1aaa-4ea4-a011-8091d01f6504" path="/var/lib/kubelet/pods/5e266ed1-1aaa-4ea4-a011-8091d01f6504/volumes"
Sep 08 13:34:39.311077 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.311043    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"]
Sep 08 13:34:39.311493 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.311405    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65c770f3-41b0-4863-8052-fb44b468fd77" containerName="main"
Sep 08 13:34:39.311493 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.311417    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="65c770f3-41b0-4863-8052-fb44b468fd77" containerName="main"
Sep 08 13:34:39.311493 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.311429    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e266ed1-1aaa-4ea4-a011-8091d01f6504" containerName="istio-proxy"
Sep 08 13:34:39.311493 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.311434    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e266ed1-1aaa-4ea4-a011-8091d01f6504" containerName="istio-proxy"
Sep 08 13:34:39.311651 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.311502    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e266ed1-1aaa-4ea4-a011-8091d01f6504" containerName="istio-proxy"
Sep 08 13:34:39.311651 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.311515    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="65c770f3-41b0-4863-8052-fb44b468fd77" containerName="main"
Sep 08 13:34:39.313589 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.313567    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:39.316944 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.316923    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-trfvb\""
Sep 08 13:34:39.317079 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.316950    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 08 13:34:39.323334 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.323305    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"]
Sep 08 13:34:39.382826 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.382785    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aaaabe9-4c29-422b-a26c-6d0717f33648-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:39.382995 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.382834    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:39.382995 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.382895    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:39.382995 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.382928    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:39.382995 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.382947    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp6t4\" (UniqueName: \"kubernetes.io/projected/6aaaabe9-4c29-422b-a26c-6d0717f33648-kube-api-access-gp6t4\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:39.383138 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.383060    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:39.484470 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.484433    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aaaabe9-4c29-422b-a26c-6d0717f33648-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:39.484661 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.484487    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:39.484661 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.484527    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:39.484661 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.484550    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:39.484661 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.484575    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp6t4\" (UniqueName: \"kubernetes.io/projected/6aaaabe9-4c29-422b-a26c-6d0717f33648-kube-api-access-gp6t4\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:39.484661 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.484648    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:39.484928 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.484898    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:39.484988 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.484949    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:39.485060 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.485043    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:39.486946 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.486921    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:39.487161 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.487144    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aaaabe9-4c29-422b-a26c-6d0717f33648-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:39.492612 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.492590    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp6t4\" (UniqueName: \"kubernetes.io/projected/6aaaabe9-4c29-422b-a26c-6d0717f33648-kube-api-access-gp6t4\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:39.625665 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.625570    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:39.900878 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:39.900853    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"]
Sep 08 13:34:39.902868 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:34:39.902833    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aaaabe9_4c29_422b_a26c_6d0717f33648.slice/crio-3ded608ecba33840c99fac8b7bd24aff4472715bfddf0f02ac322c1020d2c300 WatchSource:0}: Error finding container 3ded608ecba33840c99fac8b7bd24aff4472715bfddf0f02ac322c1020d2c300: Status 404 returned error can't find the container with id 3ded608ecba33840c99fac8b7bd24aff4472715bfddf0f02ac322c1020d2c300
Sep 08 13:34:40.311949 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:40.311907    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv" event={"ID":"6aaaabe9-4c29-422b-a26c-6d0717f33648","Type":"ContainerStarted","Data":"d178f381e2f6c88ba18af58ebe177dd2c6e7c8528dd94b7d684a4c6b40627757"}
Sep 08 13:34:40.312376 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:40.311973    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv" event={"ID":"6aaaabe9-4c29-422b-a26c-6d0717f33648","Type":"ContainerStarted","Data":"3ded608ecba33840c99fac8b7bd24aff4472715bfddf0f02ac322c1020d2c300"}
Sep 08 13:34:40.332394 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:40.332339    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv" podStartSLOduration=1.332322594 podStartE2EDuration="1.332322594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:34:40.331063417 +0000 UTC m=+1111.791708228" watchObservedRunningTime="2026-09-08 13:34:40.332322594 +0000 UTC m=+1111.792967403"
Sep 08 13:34:40.626665 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:40.626579    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:43.521428 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:43.521391    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"]
Sep 08 13:34:43.530684 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:34:43.530648    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:34:43.530684 ip-10-0-130-178 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:34:43.530684 ip-10-0-130-178 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 08 13:34:43.530684 ip-10-0-130-178 kubenswrapper[2561]:  > containerID="d178f381e2f6c88ba18af58ebe177dd2c6e7c8528dd94b7d684a4c6b40627757" cmd=["/bin/sleep","15"]
Sep 08 13:34:43.530917 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:43.530699    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv" message="" err=<
Sep 08 13:34:43.530917 ip-10-0-130-178 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:34:43.530917 ip-10-0-130-178 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 08 13:34:43.530917 ip-10-0-130-178 kubenswrapper[2561]:  >
Sep 08 13:34:43.530917 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:34:43.530711    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:34:43.530917 ip-10-0-130-178 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:34:43.530917 ip-10-0-130-178 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 08 13:34:43.530917 ip-10-0-130-178 kubenswrapper[2561]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv" podUID="6aaaabe9-4c29-422b-a26c-6d0717f33648" containerName="main" containerID="cri-o://d178f381e2f6c88ba18af58ebe177dd2c6e7c8528dd94b7d684a4c6b40627757"
Sep 08 13:34:43.530917 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:43.530746    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv" podUID="6aaaabe9-4c29-422b-a26c-6d0717f33648" containerName="main" containerID="cri-o://d178f381e2f6c88ba18af58ebe177dd2c6e7c8528dd94b7d684a4c6b40627757" gracePeriod=60
Sep 08 13:34:43.820660 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:43.820636    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:43.925913 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:43.925877    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-tmp-dir\") pod \"6aaaabe9-4c29-422b-a26c-6d0717f33648\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") "
Sep 08 13:34:43.926084 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:43.925949    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-dshm\") pod \"6aaaabe9-4c29-422b-a26c-6d0717f33648\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") "
Sep 08 13:34:43.926084 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:43.925987    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-home\") pod \"6aaaabe9-4c29-422b-a26c-6d0717f33648\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") "
Sep 08 13:34:43.926084 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:43.926017    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp6t4\" (UniqueName: \"kubernetes.io/projected/6aaaabe9-4c29-422b-a26c-6d0717f33648-kube-api-access-gp6t4\") pod \"6aaaabe9-4c29-422b-a26c-6d0717f33648\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") "
Sep 08 13:34:43.926215 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:43.926084    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aaaabe9-4c29-422b-a26c-6d0717f33648-tls-certs\") pod \"6aaaabe9-4c29-422b-a26c-6d0717f33648\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") "
Sep 08 13:34:43.926215 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:43.926115    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-model-cache\") pod \"6aaaabe9-4c29-422b-a26c-6d0717f33648\" (UID: \"6aaaabe9-4c29-422b-a26c-6d0717f33648\") "
Sep 08 13:34:43.926353 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:43.926208    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6aaaabe9-4c29-422b-a26c-6d0717f33648" (UID: "6aaaabe9-4c29-422b-a26c-6d0717f33648"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:34:43.926353 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:43.926230    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-home" (OuterVolumeSpecName: "home") pod "6aaaabe9-4c29-422b-a26c-6d0717f33648" (UID: "6aaaabe9-4c29-422b-a26c-6d0717f33648"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:34:43.926492 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:43.926460    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-model-cache" (OuterVolumeSpecName: "model-cache") pod "6aaaabe9-4c29-422b-a26c-6d0717f33648" (UID: "6aaaabe9-4c29-422b-a26c-6d0717f33648"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:34:43.926548 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:43.926500    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-tmp-dir\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:43.926548 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:43.926519    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-home\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:43.928325 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:43.928275    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aaaabe9-4c29-422b-a26c-6d0717f33648-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6aaaabe9-4c29-422b-a26c-6d0717f33648" (UID: "6aaaabe9-4c29-422b-a26c-6d0717f33648"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:34:43.928599 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:43.928580    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-dshm" (OuterVolumeSpecName: "dshm") pod "6aaaabe9-4c29-422b-a26c-6d0717f33648" (UID: "6aaaabe9-4c29-422b-a26c-6d0717f33648"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:34:43.928651 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:43.928633    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aaaabe9-4c29-422b-a26c-6d0717f33648-kube-api-access-gp6t4" (OuterVolumeSpecName: "kube-api-access-gp6t4") pod "6aaaabe9-4c29-422b-a26c-6d0717f33648" (UID: "6aaaabe9-4c29-422b-a26c-6d0717f33648"). InnerVolumeSpecName "kube-api-access-gp6t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:34:44.027975 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:44.027934    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aaaabe9-4c29-422b-a26c-6d0717f33648-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:44.027975 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:44.027967    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-model-cache\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:44.027975 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:44.027976    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6aaaabe9-4c29-422b-a26c-6d0717f33648-dshm\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:44.027975 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:44.027985    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp6t4\" (UniqueName: \"kubernetes.io/projected/6aaaabe9-4c29-422b-a26c-6d0717f33648-kube-api-access-gp6t4\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:44.336380 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:44.336345    2561 generic.go:356] "Generic (PLEG): container finished" podID="6aaaabe9-4c29-422b-a26c-6d0717f33648" containerID="d178f381e2f6c88ba18af58ebe177dd2c6e7c8528dd94b7d684a4c6b40627757" exitCode=0
Sep 08 13:34:44.336556 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:44.336394    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"
Sep 08 13:34:44.336556 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:44.336428    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv" event={"ID":"6aaaabe9-4c29-422b-a26c-6d0717f33648","Type":"ContainerDied","Data":"d178f381e2f6c88ba18af58ebe177dd2c6e7c8528dd94b7d684a4c6b40627757"}
Sep 08 13:34:44.336556 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:44.336475    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv" event={"ID":"6aaaabe9-4c29-422b-a26c-6d0717f33648","Type":"ContainerDied","Data":"3ded608ecba33840c99fac8b7bd24aff4472715bfddf0f02ac322c1020d2c300"}
Sep 08 13:34:44.336556 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:44.336500    2561 scope.go:117] "RemoveContainer" containerID="d178f381e2f6c88ba18af58ebe177dd2c6e7c8528dd94b7d684a4c6b40627757"
Sep 08 13:34:44.349526 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:44.349225    2561 scope.go:117] "RemoveContainer" containerID="d178f381e2f6c88ba18af58ebe177dd2c6e7c8528dd94b7d684a4c6b40627757"
Sep 08 13:34:44.350059 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:34:44.349981    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d178f381e2f6c88ba18af58ebe177dd2c6e7c8528dd94b7d684a4c6b40627757\": container with ID starting with d178f381e2f6c88ba18af58ebe177dd2c6e7c8528dd94b7d684a4c6b40627757 not found: ID does not exist" containerID="d178f381e2f6c88ba18af58ebe177dd2c6e7c8528dd94b7d684a4c6b40627757"
Sep 08 13:34:44.350059 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:44.350021    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d178f381e2f6c88ba18af58ebe177dd2c6e7c8528dd94b7d684a4c6b40627757"} err="failed to get container status \"d178f381e2f6c88ba18af58ebe177dd2c6e7c8528dd94b7d684a4c6b40627757\": rpc error: code = NotFound desc = could not find container \"d178f381e2f6c88ba18af58ebe177dd2c6e7c8528dd94b7d684a4c6b40627757\": container with ID starting with d178f381e2f6c88ba18af58ebe177dd2c6e7c8528dd94b7d684a4c6b40627757 not found: ID does not exist"
Sep 08 13:34:44.385677 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:44.385629    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"]
Sep 08 13:34:44.387891 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:44.387867    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfwffcv"]
Sep 08 13:34:45.007560 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:45.007528    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aaaabe9-4c29-422b-a26c-6d0717f33648" path="/var/lib/kubelet/pods/6aaaabe9-4c29-422b-a26c-6d0717f33648/volumes"
Sep 08 13:34:47.145154 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:47.145111    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"]
Sep 08 13:34:47.145575 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:47.145398    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6" podUID="40d9640d-8980-41f5-a657-4286e85ee4fb" containerName="istio-proxy" containerID="cri-o://1e70bb1cb24636cc1729fa8edf95f40a584d91822352b9c4ed628be3f855fdb9" gracePeriod=30
Sep 08 13:34:49.299153 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.299130    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:49.368714 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.368683    2561 generic.go:356] "Generic (PLEG): container finished" podID="40d9640d-8980-41f5-a657-4286e85ee4fb" containerID="1e70bb1cb24636cc1729fa8edf95f40a584d91822352b9c4ed628be3f855fdb9" exitCode=0
Sep 08 13:34:49.368897 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.368739    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"
Sep 08 13:34:49.368897 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.368766    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6" event={"ID":"40d9640d-8980-41f5-a657-4286e85ee4fb","Type":"ContainerDied","Data":"1e70bb1cb24636cc1729fa8edf95f40a584d91822352b9c4ed628be3f855fdb9"}
Sep 08 13:34:49.368897 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.368817    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6" event={"ID":"40d9640d-8980-41f5-a657-4286e85ee4fb","Type":"ContainerDied","Data":"dde83503209cf643893a1bd542157f9ad772e6c7cecd7e5bb7f4921c471a2c22"}
Sep 08 13:34:49.368897 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.368839    2561 scope.go:117] "RemoveContainer" containerID="1e70bb1cb24636cc1729fa8edf95f40a584d91822352b9c4ed628be3f855fdb9"
Sep 08 13:34:49.378277 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.378254    2561 scope.go:117] "RemoveContainer" containerID="1e70bb1cb24636cc1729fa8edf95f40a584d91822352b9c4ed628be3f855fdb9"
Sep 08 13:34:49.378602 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:34:49.378581    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e70bb1cb24636cc1729fa8edf95f40a584d91822352b9c4ed628be3f855fdb9\": container with ID starting with 1e70bb1cb24636cc1729fa8edf95f40a584d91822352b9c4ed628be3f855fdb9 not found: ID does not exist" containerID="1e70bb1cb24636cc1729fa8edf95f40a584d91822352b9c4ed628be3f855fdb9"
Sep 08 13:34:49.378700 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.378610    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e70bb1cb24636cc1729fa8edf95f40a584d91822352b9c4ed628be3f855fdb9"} err="failed to get container status \"1e70bb1cb24636cc1729fa8edf95f40a584d91822352b9c4ed628be3f855fdb9\": rpc error: code = NotFound desc = could not find container \"1e70bb1cb24636cc1729fa8edf95f40a584d91822352b9c4ed628be3f855fdb9\": container with ID starting with 1e70bb1cb24636cc1729fa8edf95f40a584d91822352b9c4ed628be3f855fdb9 not found: ID does not exist"
Sep 08 13:34:49.379797 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.379782    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bhbj\" (UniqueName: \"kubernetes.io/projected/40d9640d-8980-41f5-a657-4286e85ee4fb-kube-api-access-6bhbj\") pod \"40d9640d-8980-41f5-a657-4286e85ee4fb\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") "
Sep 08 13:34:49.379897 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.379824    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-podinfo\") pod \"40d9640d-8980-41f5-a657-4286e85ee4fb\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") "
Sep 08 13:34:49.379897 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.379855    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-credential-socket\") pod \"40d9640d-8980-41f5-a657-4286e85ee4fb\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") "
Sep 08 13:34:49.380021 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.379897    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-data\") pod \"40d9640d-8980-41f5-a657-4286e85ee4fb\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") "
Sep 08 13:34:49.380021 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.379929    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d9640d-8980-41f5-a657-4286e85ee4fb-istiod-ca-cert\") pod \"40d9640d-8980-41f5-a657-4286e85ee4fb\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") "
Sep 08 13:34:49.380021 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.379951    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-token\") pod \"40d9640d-8980-41f5-a657-4286e85ee4fb\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") "
Sep 08 13:34:49.380021 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.379985    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-workload-socket\") pod \"40d9640d-8980-41f5-a657-4286e85ee4fb\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") "
Sep 08 13:34:49.380021 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.380007    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-workload-certs\") pod \"40d9640d-8980-41f5-a657-4286e85ee4fb\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") "
Sep 08 13:34:49.380702 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.380087    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-envoy\") pod \"40d9640d-8980-41f5-a657-4286e85ee4fb\" (UID: \"40d9640d-8980-41f5-a657-4286e85ee4fb\") "
Sep 08 13:34:49.380702 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.380202    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "40d9640d-8980-41f5-a657-4286e85ee4fb" (UID: "40d9640d-8980-41f5-a657-4286e85ee4fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:34:49.380702 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.380303    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-data" (OuterVolumeSpecName: "istio-data") pod "40d9640d-8980-41f5-a657-4286e85ee4fb" (UID: "40d9640d-8980-41f5-a657-4286e85ee4fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:34:49.380702 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.380309    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d9640d-8980-41f5-a657-4286e85ee4fb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "40d9640d-8980-41f5-a657-4286e85ee4fb" (UID: "40d9640d-8980-41f5-a657-4286e85ee4fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:34:49.380702 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.380455    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-data\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:49.380702 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.380476    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d9640d-8980-41f5-a657-4286e85ee4fb-istiod-ca-cert\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:49.380702 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.380489    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-credential-socket\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:49.380702 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.380648    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "40d9640d-8980-41f5-a657-4286e85ee4fb" (UID: "40d9640d-8980-41f5-a657-4286e85ee4fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:34:49.381126 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.380708    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "40d9640d-8980-41f5-a657-4286e85ee4fb" (UID: "40d9640d-8980-41f5-a657-4286e85ee4fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:34:49.382202 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.382178    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "40d9640d-8980-41f5-a657-4286e85ee4fb" (UID: "40d9640d-8980-41f5-a657-4286e85ee4fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 13:34:49.382533 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.382506    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-token" (OuterVolumeSpecName: "istio-token") pod "40d9640d-8980-41f5-a657-4286e85ee4fb" (UID: "40d9640d-8980-41f5-a657-4286e85ee4fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:34:49.382681 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.382662    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d9640d-8980-41f5-a657-4286e85ee4fb-kube-api-access-6bhbj" (OuterVolumeSpecName: "kube-api-access-6bhbj") pod "40d9640d-8980-41f5-a657-4286e85ee4fb" (UID: "40d9640d-8980-41f5-a657-4286e85ee4fb"). InnerVolumeSpecName "kube-api-access-6bhbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:34:49.382741 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.382705    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "40d9640d-8980-41f5-a657-4286e85ee4fb" (UID: "40d9640d-8980-41f5-a657-4286e85ee4fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:34:49.481481 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.481447    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-token\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:49.481481 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.481474    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-workload-socket\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:49.481481 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.481484    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-workload-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:49.481705 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.481494    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-envoy\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:49.481705 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.481503    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bhbj\" (UniqueName: \"kubernetes.io/projected/40d9640d-8980-41f5-a657-4286e85ee4fb-kube-api-access-6bhbj\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:49.481705 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.481512    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d9640d-8980-41f5-a657-4286e85ee4fb-istio-podinfo\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:34:49.690595 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.690565    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"]
Sep 08 13:34:49.694430 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:49.694399    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-t5mt6"]
Sep 08 13:34:51.007174 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:51.007139    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40d9640d-8980-41f5-a657-4286e85ee4fb" path="/var/lib/kubelet/pods/40d9640d-8980-41f5-a657-4286e85ee4fb/volumes"
Sep 08 13:34:58.051089 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.051034    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv"]
Sep 08 13:34:58.051650 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.051627    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aaaabe9-4c29-422b-a26c-6d0717f33648" containerName="main"
Sep 08 13:34:58.051727 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.051654    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aaaabe9-4c29-422b-a26c-6d0717f33648" containerName="main"
Sep 08 13:34:58.051727 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.051681    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d9640d-8980-41f5-a657-4286e85ee4fb" containerName="istio-proxy"
Sep 08 13:34:58.051727 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.051690    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d9640d-8980-41f5-a657-4286e85ee4fb" containerName="istio-proxy"
Sep 08 13:34:58.052364 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.051783    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="40d9640d-8980-41f5-a657-4286e85ee4fb" containerName="istio-proxy"
Sep 08 13:34:58.052364 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.051799    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aaaabe9-4c29-422b-a26c-6d0717f33648" containerName="main"
Sep 08 13:34:58.055603 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.055577    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv"
Sep 08 13:34:58.059012 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.058987    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-8vx65\""
Sep 08 13:34:58.059244 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.059002    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 08 13:34:58.059377 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.059003    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-xwbb4\""
Sep 08 13:34:58.059377 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.059011    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 08 13:34:58.059377 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.059037    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 08 13:34:58.066636 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.066613    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv"]
Sep 08 13:34:58.159454 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.159406    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmfhw\" (UniqueName: \"kubernetes.io/projected/f38aee75-a82f-4317-add4-d5dc1d0c2653-kube-api-access-fmfhw\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv\" (UID: \"f38aee75-a82f-4317-add4-d5dc1d0c2653\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv"
Sep 08 13:34:58.159641 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.159616    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f38aee75-a82f-4317-add4-d5dc1d0c2653-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv\" (UID: \"f38aee75-a82f-4317-add4-d5dc1d0c2653\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv"
Sep 08 13:34:58.260371 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.260331    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f38aee75-a82f-4317-add4-d5dc1d0c2653-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv\" (UID: \"f38aee75-a82f-4317-add4-d5dc1d0c2653\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv"
Sep 08 13:34:58.260553 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.260399    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmfhw\" (UniqueName: \"kubernetes.io/projected/f38aee75-a82f-4317-add4-d5dc1d0c2653-kube-api-access-fmfhw\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv\" (UID: \"f38aee75-a82f-4317-add4-d5dc1d0c2653\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv"
Sep 08 13:34:58.263206 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.263179    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f38aee75-a82f-4317-add4-d5dc1d0c2653-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv\" (UID: \"f38aee75-a82f-4317-add4-d5dc1d0c2653\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv"
Sep 08 13:34:58.268862 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.268837    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmfhw\" (UniqueName: \"kubernetes.io/projected/f38aee75-a82f-4317-add4-d5dc1d0c2653-kube-api-access-fmfhw\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv\" (UID: \"f38aee75-a82f-4317-add4-d5dc1d0c2653\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv"
Sep 08 13:34:58.370070 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.369978    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv"
Sep 08 13:34:58.648020 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:58.647978    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv"]
Sep 08 13:34:58.651011 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:34:58.650981    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38aee75_a82f_4317_add4_d5dc1d0c2653.slice/crio-4d4ff28a49c5b5bbcca15dbe9b199a3850112c63b492b707aebf4573abe581bc WatchSource:0}: Error finding container 4d4ff28a49c5b5bbcca15dbe9b199a3850112c63b492b707aebf4573abe581bc: Status 404 returned error can't find the container with id 4d4ff28a49c5b5bbcca15dbe9b199a3850112c63b492b707aebf4573abe581bc
Sep 08 13:34:59.426200 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:59.426167    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv" event={"ID":"f38aee75-a82f-4317-add4-d5dc1d0c2653","Type":"ContainerStarted","Data":"e439fc06b87026930fd58958f9367e3db0d348f723cb2360abc1e1ae391f33b3"}
Sep 08 13:34:59.426701 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:59.426218    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv" event={"ID":"f38aee75-a82f-4317-add4-d5dc1d0c2653","Type":"ContainerStarted","Data":"4d4ff28a49c5b5bbcca15dbe9b199a3850112c63b492b707aebf4573abe581bc"}
Sep 08 13:34:59.426701 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:59.426322    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv"
Sep 08 13:34:59.445131 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:34:59.445064    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv" podStartSLOduration=1.4450437059999999 podStartE2EDuration="1.445043706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:34:59.444667624 +0000 UTC m=+1130.905312436" watchObservedRunningTime="2026-09-08 13:34:59.445043706 +0000 UTC m=+1130.905688517"
Sep 08 13:35:30.433675 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:35:30.433640    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv"
Sep 08 13:36:09.024021 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:09.023989    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 13:36:09.025036 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:09.025014    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 13:36:15.608609 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:15.608576    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b"]
Sep 08 13:36:15.611158 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:15.611139    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b"
Sep 08 13:36:15.614464 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:15.614435    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 08 13:36:15.614771 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:15.614752    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-mx8zh\""
Sep 08 13:36:15.615353 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:15.615330    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-l665j\""
Sep 08 13:36:15.615427 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:15.615338    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 08 13:36:15.620662 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:15.620642    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 08 13:36:15.628912 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:15.628888    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b"]
Sep 08 13:36:15.754283 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:15.754248    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brw78\" (UniqueName: \"kubernetes.io/projected/c42e7b7b-8b84-42f5-aa35-c76a30113ce8-kube-api-access-brw78\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b\" (UID: \"c42e7b7b-8b84-42f5-aa35-c76a30113ce8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b"
Sep 08 13:36:15.754484 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:15.754382    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42e7b7b-8b84-42f5-aa35-c76a30113ce8-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b\" (UID: \"c42e7b7b-8b84-42f5-aa35-c76a30113ce8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b"
Sep 08 13:36:15.855738 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:15.855700    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42e7b7b-8b84-42f5-aa35-c76a30113ce8-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b\" (UID: \"c42e7b7b-8b84-42f5-aa35-c76a30113ce8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b"
Sep 08 13:36:15.855942 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:15.855779    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brw78\" (UniqueName: \"kubernetes.io/projected/c42e7b7b-8b84-42f5-aa35-c76a30113ce8-kube-api-access-brw78\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b\" (UID: \"c42e7b7b-8b84-42f5-aa35-c76a30113ce8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b"
Sep 08 13:36:15.858260 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:15.858231    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42e7b7b-8b84-42f5-aa35-c76a30113ce8-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b\" (UID: \"c42e7b7b-8b84-42f5-aa35-c76a30113ce8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b"
Sep 08 13:36:15.863996 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:15.863927    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brw78\" (UniqueName: \"kubernetes.io/projected/c42e7b7b-8b84-42f5-aa35-c76a30113ce8-kube-api-access-brw78\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b\" (UID: \"c42e7b7b-8b84-42f5-aa35-c76a30113ce8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b"
Sep 08 13:36:15.922462 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:15.922410    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b"
Sep 08 13:36:16.194052 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:16.194025    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b"]
Sep 08 13:36:16.196383 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:36:16.196346    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42e7b7b_8b84_42f5_aa35_c76a30113ce8.slice/crio-69328108b9a0c6115479aedfeb06df98ad24b3dc1dbeec9c721362b5bafabae4 WatchSource:0}: Error finding container 69328108b9a0c6115479aedfeb06df98ad24b3dc1dbeec9c721362b5bafabae4: Status 404 returned error can't find the container with id 69328108b9a0c6115479aedfeb06df98ad24b3dc1dbeec9c721362b5bafabae4
Sep 08 13:36:16.845789 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:16.845748    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b" event={"ID":"c42e7b7b-8b84-42f5-aa35-c76a30113ce8","Type":"ContainerStarted","Data":"91d69f263a2d942be47065c560e6a1de7178f242060b0c2ce6937aba147091dc"}
Sep 08 13:36:16.846228 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:16.845807    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b" event={"ID":"c42e7b7b-8b84-42f5-aa35-c76a30113ce8","Type":"ContainerStarted","Data":"69328108b9a0c6115479aedfeb06df98ad24b3dc1dbeec9c721362b5bafabae4"}
Sep 08 13:36:16.846228 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:16.845871    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b"
Sep 08 13:36:16.871334 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:16.871264    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b" podStartSLOduration=1.8712485540000001 podStartE2EDuration="1.871248554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:36:16.868666489 +0000 UTC m=+1208.329311296" watchObservedRunningTime="2026-09-08 13:36:16.871248554 +0000 UTC m=+1208.331893364"
Sep 08 13:36:47.853885 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:47.853847    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b"
Sep 08 13:36:49.899675 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:49.899635    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b"]
Sep 08 13:36:57.853819 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:36:57.853783    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b" podUID="c42e7b7b-8b84-42f5-aa35-c76a30113ce8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:37:03.275511 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:03.275424    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv"]
Sep 08 13:37:04.926014 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:04.925972    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b" podUID="c42e7b7b-8b84-42f5-aa35-c76a30113ce8" containerName="main" containerID="cri-o://91d69f263a2d942be47065c560e6a1de7178f242060b0c2ce6937aba147091dc" gracePeriod=45
Sep 08 13:37:07.854013 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:07.853970    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b" podUID="c42e7b7b-8b84-42f5-aa35-c76a30113ce8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:37:10.433181 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:10.433142    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv" podUID="f38aee75-a82f-4317-add4-d5dc1d0c2653" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:37:17.853767 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:17.853728    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b" podUID="c42e7b7b-8b84-42f5-aa35-c76a30113ce8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:37:17.854247 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:17.853826    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b"
Sep 08 13:37:18.297215 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:18.297169    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv" podUID="f38aee75-a82f-4317-add4-d5dc1d0c2653" containerName="main" containerID="cri-o://e439fc06b87026930fd58958f9367e3db0d348f723cb2360abc1e1ae391f33b3" gracePeriod=45
Sep 08 13:37:20.433965 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:20.433921    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv" podUID="f38aee75-a82f-4317-add4-d5dc1d0c2653" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:37:27.853630 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:27.853587    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b" podUID="c42e7b7b-8b84-42f5-aa35-c76a30113ce8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:37:30.433942 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:30.433905    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv" podUID="f38aee75-a82f-4317-add4-d5dc1d0c2653" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:37:30.434408 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:30.434039    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv"
Sep 08 13:37:35.180043 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:35.180015    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b"
Sep 08 13:37:35.269747 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:35.269655    2561 generic.go:356] "Generic (PLEG): container finished" podID="c42e7b7b-8b84-42f5-aa35-c76a30113ce8" containerID="91d69f263a2d942be47065c560e6a1de7178f242060b0c2ce6937aba147091dc" exitCode=0
Sep 08 13:37:35.269747 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:35.269700    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b"
Sep 08 13:37:35.269747 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:35.269732    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b" event={"ID":"c42e7b7b-8b84-42f5-aa35-c76a30113ce8","Type":"ContainerDied","Data":"91d69f263a2d942be47065c560e6a1de7178f242060b0c2ce6937aba147091dc"}
Sep 08 13:37:35.270011 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:35.269786    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b" event={"ID":"c42e7b7b-8b84-42f5-aa35-c76a30113ce8","Type":"ContainerDied","Data":"69328108b9a0c6115479aedfeb06df98ad24b3dc1dbeec9c721362b5bafabae4"}
Sep 08 13:37:35.270011 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:35.269811    2561 scope.go:117] "RemoveContainer" containerID="91d69f263a2d942be47065c560e6a1de7178f242060b0c2ce6937aba147091dc"
Sep 08 13:37:35.270781 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:35.270760    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brw78\" (UniqueName: \"kubernetes.io/projected/c42e7b7b-8b84-42f5-aa35-c76a30113ce8-kube-api-access-brw78\") pod \"c42e7b7b-8b84-42f5-aa35-c76a30113ce8\" (UID: \"c42e7b7b-8b84-42f5-aa35-c76a30113ce8\") "
Sep 08 13:37:35.270925 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:35.270852    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42e7b7b-8b84-42f5-aa35-c76a30113ce8-tls-certs\") pod \"c42e7b7b-8b84-42f5-aa35-c76a30113ce8\" (UID: \"c42e7b7b-8b84-42f5-aa35-c76a30113ce8\") "
Sep 08 13:37:35.273173 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:35.273143    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42e7b7b-8b84-42f5-aa35-c76a30113ce8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c42e7b7b-8b84-42f5-aa35-c76a30113ce8" (UID: "c42e7b7b-8b84-42f5-aa35-c76a30113ce8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:37:35.273173 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:35.273149    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42e7b7b-8b84-42f5-aa35-c76a30113ce8-kube-api-access-brw78" (OuterVolumeSpecName: "kube-api-access-brw78") pod "c42e7b7b-8b84-42f5-aa35-c76a30113ce8" (UID: "c42e7b7b-8b84-42f5-aa35-c76a30113ce8"). InnerVolumeSpecName "kube-api-access-brw78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:37:35.284788 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:35.284765    2561 scope.go:117] "RemoveContainer" containerID="91d69f263a2d942be47065c560e6a1de7178f242060b0c2ce6937aba147091dc"
Sep 08 13:37:35.285144 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:37:35.285122    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d69f263a2d942be47065c560e6a1de7178f242060b0c2ce6937aba147091dc\": container with ID starting with 91d69f263a2d942be47065c560e6a1de7178f242060b0c2ce6937aba147091dc not found: ID does not exist" containerID="91d69f263a2d942be47065c560e6a1de7178f242060b0c2ce6937aba147091dc"
Sep 08 13:37:35.285183 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:35.285155    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d69f263a2d942be47065c560e6a1de7178f242060b0c2ce6937aba147091dc"} err="failed to get container status \"91d69f263a2d942be47065c560e6a1de7178f242060b0c2ce6937aba147091dc\": rpc error: code = NotFound desc = could not find container \"91d69f263a2d942be47065c560e6a1de7178f242060b0c2ce6937aba147091dc\": container with ID starting with 91d69f263a2d942be47065c560e6a1de7178f242060b0c2ce6937aba147091dc not found: ID does not exist"
Sep 08 13:37:35.372007 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:35.371965    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brw78\" (UniqueName: \"kubernetes.io/projected/c42e7b7b-8b84-42f5-aa35-c76a30113ce8-kube-api-access-brw78\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:37:35.372007 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:35.372001    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42e7b7b-8b84-42f5-aa35-c76a30113ce8-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:37:35.593417 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:35.593388    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b"]
Sep 08 13:37:35.597807 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:35.597778    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbvs5b"]
Sep 08 13:37:37.006148 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:37.006116    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c42e7b7b-8b84-42f5-aa35-c76a30113ce8" path="/var/lib/kubelet/pods/c42e7b7b-8b84-42f5-aa35-c76a30113ce8/volumes"
Sep 08 13:37:40.433755 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:40.433718    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv" podUID="f38aee75-a82f-4317-add4-d5dc1d0c2653" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:37:48.543018 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:48.542985    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv"
Sep 08 13:37:48.689434 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:48.689398    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f38aee75-a82f-4317-add4-d5dc1d0c2653-tls-certs\") pod \"f38aee75-a82f-4317-add4-d5dc1d0c2653\" (UID: \"f38aee75-a82f-4317-add4-d5dc1d0c2653\") "
Sep 08 13:37:48.689606 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:48.689503    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmfhw\" (UniqueName: \"kubernetes.io/projected/f38aee75-a82f-4317-add4-d5dc1d0c2653-kube-api-access-fmfhw\") pod \"f38aee75-a82f-4317-add4-d5dc1d0c2653\" (UID: \"f38aee75-a82f-4317-add4-d5dc1d0c2653\") "
Sep 08 13:37:48.691766 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:48.691740    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f38aee75-a82f-4317-add4-d5dc1d0c2653-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f38aee75-a82f-4317-add4-d5dc1d0c2653" (UID: "f38aee75-a82f-4317-add4-d5dc1d0c2653"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:37:48.691859 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:48.691740    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38aee75-a82f-4317-add4-d5dc1d0c2653-kube-api-access-fmfhw" (OuterVolumeSpecName: "kube-api-access-fmfhw") pod "f38aee75-a82f-4317-add4-d5dc1d0c2653" (UID: "f38aee75-a82f-4317-add4-d5dc1d0c2653"). InnerVolumeSpecName "kube-api-access-fmfhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:37:48.790923 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:48.790879    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmfhw\" (UniqueName: \"kubernetes.io/projected/f38aee75-a82f-4317-add4-d5dc1d0c2653-kube-api-access-fmfhw\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:37:48.790923 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:48.790916    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f38aee75-a82f-4317-add4-d5dc1d0c2653-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:37:49.346928 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:49.346832    2561 generic.go:356] "Generic (PLEG): container finished" podID="f38aee75-a82f-4317-add4-d5dc1d0c2653" containerID="e439fc06b87026930fd58958f9367e3db0d348f723cb2360abc1e1ae391f33b3" exitCode=0
Sep 08 13:37:49.346928 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:49.346899    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv"
Sep 08 13:37:49.347109 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:49.346924    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv" event={"ID":"f38aee75-a82f-4317-add4-d5dc1d0c2653","Type":"ContainerDied","Data":"e439fc06b87026930fd58958f9367e3db0d348f723cb2360abc1e1ae391f33b3"}
Sep 08 13:37:49.347109 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:49.346981    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv" event={"ID":"f38aee75-a82f-4317-add4-d5dc1d0c2653","Type":"ContainerDied","Data":"4d4ff28a49c5b5bbcca15dbe9b199a3850112c63b492b707aebf4573abe581bc"}
Sep 08 13:37:49.347109 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:49.347003    2561 scope.go:117] "RemoveContainer" containerID="e439fc06b87026930fd58958f9367e3db0d348f723cb2360abc1e1ae391f33b3"
Sep 08 13:37:49.356665 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:49.356647    2561 scope.go:117] "RemoveContainer" containerID="e439fc06b87026930fd58958f9367e3db0d348f723cb2360abc1e1ae391f33b3"
Sep 08 13:37:49.356966 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:37:49.356944    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e439fc06b87026930fd58958f9367e3db0d348f723cb2360abc1e1ae391f33b3\": container with ID starting with e439fc06b87026930fd58958f9367e3db0d348f723cb2360abc1e1ae391f33b3 not found: ID does not exist" containerID="e439fc06b87026930fd58958f9367e3db0d348f723cb2360abc1e1ae391f33b3"
Sep 08 13:37:49.357032 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:49.356974    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e439fc06b87026930fd58958f9367e3db0d348f723cb2360abc1e1ae391f33b3"} err="failed to get container status \"e439fc06b87026930fd58958f9367e3db0d348f723cb2360abc1e1ae391f33b3\": rpc error: code = NotFound desc = could not find container \"e439fc06b87026930fd58958f9367e3db0d348f723cb2360abc1e1ae391f33b3\": container with ID starting with e439fc06b87026930fd58958f9367e3db0d348f723cb2360abc1e1ae391f33b3 not found: ID does not exist"
Sep 08 13:37:49.369495 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:49.369469    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv"]
Sep 08 13:37:49.374983 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:49.374962    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-5v4dv"]
Sep 08 13:37:51.007319 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:37:51.007269    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f38aee75-a82f-4317-add4-d5dc1d0c2653" path="/var/lib/kubelet/pods/f38aee75-a82f-4317-add4-d5dc1d0c2653/volumes"
Sep 08 13:38:01.662444 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:01.662404    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd"]
Sep 08 13:38:01.663255 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:01.663224    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f38aee75-a82f-4317-add4-d5dc1d0c2653" containerName="main"
Sep 08 13:38:01.663255 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:01.663251    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f38aee75-a82f-4317-add4-d5dc1d0c2653" containerName="main"
Sep 08 13:38:01.663482 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:01.663281    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c42e7b7b-8b84-42f5-aa35-c76a30113ce8" containerName="main"
Sep 08 13:38:01.663482 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:01.663308    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c42e7b7b-8b84-42f5-aa35-c76a30113ce8" containerName="main"
Sep 08 13:38:01.663482 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:01.663409    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c42e7b7b-8b84-42f5-aa35-c76a30113ce8" containerName="main"
Sep 08 13:38:01.663482 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:01.663428    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f38aee75-a82f-4317-add4-d5dc1d0c2653" containerName="main"
Sep 08 13:38:01.666618 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:01.666593    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd"
Sep 08 13:38:01.669021 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:01.668998    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-d6l9g\""
Sep 08 13:38:01.669162 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:01.669025    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 08 13:38:01.669558 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:01.669538    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 08 13:38:01.669649 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:01.669557    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-4jwjl\""
Sep 08 13:38:01.669649 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:01.669539    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 08 13:38:01.676527 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:01.676503    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd"]
Sep 08 13:38:01.808458 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:01.808419    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61b11f8-9e06-48fe-a906-b1f78bdf766d-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd\" (UID: \"c61b11f8-9e06-48fe-a906-b1f78bdf766d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd"
Sep 08 13:38:01.808640 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:01.808496    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gzhb\" (UniqueName: \"kubernetes.io/projected/c61b11f8-9e06-48fe-a906-b1f78bdf766d-kube-api-access-6gzhb\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd\" (UID: \"c61b11f8-9e06-48fe-a906-b1f78bdf766d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd"
Sep 08 13:38:01.909326 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:01.909272    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gzhb\" (UniqueName: \"kubernetes.io/projected/c61b11f8-9e06-48fe-a906-b1f78bdf766d-kube-api-access-6gzhb\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd\" (UID: \"c61b11f8-9e06-48fe-a906-b1f78bdf766d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd"
Sep 08 13:38:01.909498 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:01.909388    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61b11f8-9e06-48fe-a906-b1f78bdf766d-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd\" (UID: \"c61b11f8-9e06-48fe-a906-b1f78bdf766d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd"
Sep 08 13:38:01.911961 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:01.911933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61b11f8-9e06-48fe-a906-b1f78bdf766d-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd\" (UID: \"c61b11f8-9e06-48fe-a906-b1f78bdf766d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd"
Sep 08 13:38:01.917650 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:01.917591    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gzhb\" (UniqueName: \"kubernetes.io/projected/c61b11f8-9e06-48fe-a906-b1f78bdf766d-kube-api-access-6gzhb\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd\" (UID: \"c61b11f8-9e06-48fe-a906-b1f78bdf766d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd"
Sep 08 13:38:01.979206 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:01.979169    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd"
Sep 08 13:38:02.276234 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:02.276208    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd"]
Sep 08 13:38:02.278833 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:38:02.278803    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61b11f8_9e06_48fe_a906_b1f78bdf766d.slice/crio-5b5dbb51a7352902e7986741eb86ae5f593d890e4a1fe615de8779345ff64998 WatchSource:0}: Error finding container 5b5dbb51a7352902e7986741eb86ae5f593d890e4a1fe615de8779345ff64998: Status 404 returned error can't find the container with id 5b5dbb51a7352902e7986741eb86ae5f593d890e4a1fe615de8779345ff64998
Sep 08 13:38:02.414768 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:02.414735    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd" event={"ID":"c61b11f8-9e06-48fe-a906-b1f78bdf766d","Type":"ContainerStarted","Data":"75b695d0175b48ab892c9a9774a9d95d8eefd5269b8d02861b23c54490aee143"}
Sep 08 13:38:02.414950 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:02.414794    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd" event={"ID":"c61b11f8-9e06-48fe-a906-b1f78bdf766d","Type":"ContainerStarted","Data":"5b5dbb51a7352902e7986741eb86ae5f593d890e4a1fe615de8779345ff64998"}
Sep 08 13:38:02.414950 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:02.414823    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd"
Sep 08 13:38:02.437918 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:02.437860    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd" podStartSLOduration=1.437844015 podStartE2EDuration="1.437844015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:38:02.43658314 +0000 UTC m=+1313.897227949" watchObservedRunningTime="2026-09-08 13:38:02.437844015 +0000 UTC m=+1313.898488824"
Sep 08 13:38:14.156543 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:14.156494    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5"]
Sep 08 13:38:14.161248 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:14.161230    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5"
Sep 08 13:38:14.164264 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:14.164235    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-r2xc7\""
Sep 08 13:38:14.164264 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:14.164256    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 08 13:38:14.164479 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:14.164256    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 08 13:38:14.164479 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:14.164241    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 08 13:38:14.164643 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:14.164626    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-hstrf\""
Sep 08 13:38:14.174931 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:14.174897    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5"]
Sep 08 13:38:14.226373 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:14.226332    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnlzr\" (UniqueName: \"kubernetes.io/projected/27cf4e93-f79b-4a82-9446-23f8e5afebc4-kube-api-access-nnlzr\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5\" (UID: \"27cf4e93-f79b-4a82-9446-23f8e5afebc4\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5"
Sep 08 13:38:14.226578 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:14.226403    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27cf4e93-f79b-4a82-9446-23f8e5afebc4-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5\" (UID: \"27cf4e93-f79b-4a82-9446-23f8e5afebc4\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5"
Sep 08 13:38:14.327678 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:14.327633    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnlzr\" (UniqueName: \"kubernetes.io/projected/27cf4e93-f79b-4a82-9446-23f8e5afebc4-kube-api-access-nnlzr\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5\" (UID: \"27cf4e93-f79b-4a82-9446-23f8e5afebc4\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5"
Sep 08 13:38:14.327850 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:14.327718    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27cf4e93-f79b-4a82-9446-23f8e5afebc4-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5\" (UID: \"27cf4e93-f79b-4a82-9446-23f8e5afebc4\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5"
Sep 08 13:38:14.330398 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:14.330371    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27cf4e93-f79b-4a82-9446-23f8e5afebc4-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5\" (UID: \"27cf4e93-f79b-4a82-9446-23f8e5afebc4\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5"
Sep 08 13:38:14.336066 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:14.336039    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnlzr\" (UniqueName: \"kubernetes.io/projected/27cf4e93-f79b-4a82-9446-23f8e5afebc4-kube-api-access-nnlzr\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5\" (UID: \"27cf4e93-f79b-4a82-9446-23f8e5afebc4\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5"
Sep 08 13:38:14.472256 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:14.472213    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5"
Sep 08 13:38:14.751240 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:14.751209    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5"]
Sep 08 13:38:14.753354 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:38:14.753325    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27cf4e93_f79b_4a82_9446_23f8e5afebc4.slice/crio-f32b28be6cd961410f7d5ad973b3b7b5975306e753912ad87d4bb2180418d80b WatchSource:0}: Error finding container f32b28be6cd961410f7d5ad973b3b7b5975306e753912ad87d4bb2180418d80b: Status 404 returned error can't find the container with id f32b28be6cd961410f7d5ad973b3b7b5975306e753912ad87d4bb2180418d80b
Sep 08 13:38:15.493346 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:15.493302    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5" event={"ID":"27cf4e93-f79b-4a82-9446-23f8e5afebc4","Type":"ContainerStarted","Data":"d72b7bc6276e52e9d81c61e84a293b78bc7f31a6a62d35abc9349ca223c3b253"}
Sep 08 13:38:15.493733 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:15.493381    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5"
Sep 08 13:38:15.493733 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:15.493402    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5" event={"ID":"27cf4e93-f79b-4a82-9446-23f8e5afebc4","Type":"ContainerStarted","Data":"f32b28be6cd961410f7d5ad973b3b7b5975306e753912ad87d4bb2180418d80b"}
Sep 08 13:38:15.512087 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:15.512038    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5" podStartSLOduration=1.512021695 podStartE2EDuration="1.512021695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:38:15.510593264 +0000 UTC m=+1326.971238077" watchObservedRunningTime="2026-09-08 13:38:15.512021695 +0000 UTC m=+1326.972666504"
Sep 08 13:38:33.422702 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:33.422625    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd"
Sep 08 13:38:35.090811 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:35.090771    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd"]
Sep 08 13:38:43.422614 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:43.422573    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd" podUID="c61b11f8-9e06-48fe-a906-b1f78bdf766d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:38:46.502528 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:46.502495    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5"
Sep 08 13:38:50.116963 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:50.116909    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd" podUID="c61b11f8-9e06-48fe-a906-b1f78bdf766d" containerName="main" containerID="cri-o://75b695d0175b48ab892c9a9774a9d95d8eefd5269b8d02861b23c54490aee143" gracePeriod=45
Sep 08 13:38:53.422354 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:38:53.422314    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd" podUID="c61b11f8-9e06-48fe-a906-b1f78bdf766d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:39:03.421804 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:03.421763    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd" podUID="c61b11f8-9e06-48fe-a906-b1f78bdf766d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:39:03.422191 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:03.421869    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd"
Sep 08 13:39:13.422611 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:13.422569    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd" podUID="c61b11f8-9e06-48fe-a906-b1f78bdf766d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:39:20.370590 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:20.370557    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd"
Sep 08 13:39:20.408826 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:20.408736    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gzhb\" (UniqueName: \"kubernetes.io/projected/c61b11f8-9e06-48fe-a906-b1f78bdf766d-kube-api-access-6gzhb\") pod \"c61b11f8-9e06-48fe-a906-b1f78bdf766d\" (UID: \"c61b11f8-9e06-48fe-a906-b1f78bdf766d\") "
Sep 08 13:39:20.408997 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:20.408855    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61b11f8-9e06-48fe-a906-b1f78bdf766d-tls-certs\") pod \"c61b11f8-9e06-48fe-a906-b1f78bdf766d\" (UID: \"c61b11f8-9e06-48fe-a906-b1f78bdf766d\") "
Sep 08 13:39:20.410951 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:20.410918    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61b11f8-9e06-48fe-a906-b1f78bdf766d-kube-api-access-6gzhb" (OuterVolumeSpecName: "kube-api-access-6gzhb") pod "c61b11f8-9e06-48fe-a906-b1f78bdf766d" (UID: "c61b11f8-9e06-48fe-a906-b1f78bdf766d"). InnerVolumeSpecName "kube-api-access-6gzhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:39:20.411071 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:20.410974    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c61b11f8-9e06-48fe-a906-b1f78bdf766d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c61b11f8-9e06-48fe-a906-b1f78bdf766d" (UID: "c61b11f8-9e06-48fe-a906-b1f78bdf766d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:39:20.510542 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:20.510503    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gzhb\" (UniqueName: \"kubernetes.io/projected/c61b11f8-9e06-48fe-a906-b1f78bdf766d-kube-api-access-6gzhb\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:39:20.510542 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:20.510536    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61b11f8-9e06-48fe-a906-b1f78bdf766d-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:39:20.855975 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:20.855941    2561 generic.go:356] "Generic (PLEG): container finished" podID="c61b11f8-9e06-48fe-a906-b1f78bdf766d" containerID="75b695d0175b48ab892c9a9774a9d95d8eefd5269b8d02861b23c54490aee143" exitCode=0
Sep 08 13:39:20.856151 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:20.855987    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd"
Sep 08 13:39:20.856151 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:20.856025    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd" event={"ID":"c61b11f8-9e06-48fe-a906-b1f78bdf766d","Type":"ContainerDied","Data":"75b695d0175b48ab892c9a9774a9d95d8eefd5269b8d02861b23c54490aee143"}
Sep 08 13:39:20.856151 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:20.856078    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd" event={"ID":"c61b11f8-9e06-48fe-a906-b1f78bdf766d","Type":"ContainerDied","Data":"5b5dbb51a7352902e7986741eb86ae5f593d890e4a1fe615de8779345ff64998"}
Sep 08 13:39:20.856151 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:20.856106    2561 scope.go:117] "RemoveContainer" containerID="75b695d0175b48ab892c9a9774a9d95d8eefd5269b8d02861b23c54490aee143"
Sep 08 13:39:20.868839 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:20.868820    2561 scope.go:117] "RemoveContainer" containerID="75b695d0175b48ab892c9a9774a9d95d8eefd5269b8d02861b23c54490aee143"
Sep 08 13:39:20.869143 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:39:20.869122    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b695d0175b48ab892c9a9774a9d95d8eefd5269b8d02861b23c54490aee143\": container with ID starting with 75b695d0175b48ab892c9a9774a9d95d8eefd5269b8d02861b23c54490aee143 not found: ID does not exist" containerID="75b695d0175b48ab892c9a9774a9d95d8eefd5269b8d02861b23c54490aee143"
Sep 08 13:39:20.869204 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:20.869152    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b695d0175b48ab892c9a9774a9d95d8eefd5269b8d02861b23c54490aee143"} err="failed to get container status \"75b695d0175b48ab892c9a9774a9d95d8eefd5269b8d02861b23c54490aee143\": rpc error: code = NotFound desc = could not find container \"75b695d0175b48ab892c9a9774a9d95d8eefd5269b8d02861b23c54490aee143\": container with ID starting with 75b695d0175b48ab892c9a9774a9d95d8eefd5269b8d02861b23c54490aee143 not found: ID does not exist"
Sep 08 13:39:20.883833 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:20.883808    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd"]
Sep 08 13:39:20.892027 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:20.891999    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schej9hgd"]
Sep 08 13:39:21.007579 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:21.007546    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c61b11f8-9e06-48fe-a906-b1f78bdf766d" path="/var/lib/kubelet/pods/c61b11f8-9e06-48fe-a906-b1f78bdf766d/volumes"
Sep 08 13:39:31.998613 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:31.998569    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc"]
Sep 08 13:39:31.999083 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:31.998982    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c61b11f8-9e06-48fe-a906-b1f78bdf766d" containerName="main"
Sep 08 13:39:31.999083 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:31.998996    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c61b11f8-9e06-48fe-a906-b1f78bdf766d" containerName="main"
Sep 08 13:39:31.999200 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:31.999102    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c61b11f8-9e06-48fe-a906-b1f78bdf766d" containerName="main"
Sep 08 13:39:32.003884 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:32.003857    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc"
Sep 08 13:39:32.006492 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:32.006465    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 08 13:39:32.007063 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:32.007039    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 08 13:39:32.007364 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:32.007344    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-jjfnq\""
Sep 08 13:39:32.007533 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:32.007513    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 08 13:39:32.007623 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:32.007588    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-9zkkj\""
Sep 08 13:39:32.016653 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:32.016627    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc"]
Sep 08 13:39:32.123725 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:32.123688    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6zs6\" (UniqueName: \"kubernetes.io/projected/1da9c52e-d938-4ead-a592-15df64a66ecf-kube-api-access-x6zs6\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc\" (UID: \"1da9c52e-d938-4ead-a592-15df64a66ecf\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc"
Sep 08 13:39:32.123894 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:32.123847    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1da9c52e-d938-4ead-a592-15df64a66ecf-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc\" (UID: \"1da9c52e-d938-4ead-a592-15df64a66ecf\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc"
Sep 08 13:39:32.225172 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:32.225131    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1da9c52e-d938-4ead-a592-15df64a66ecf-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc\" (UID: \"1da9c52e-d938-4ead-a592-15df64a66ecf\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc"
Sep 08 13:39:32.225416 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:32.225228    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6zs6\" (UniqueName: \"kubernetes.io/projected/1da9c52e-d938-4ead-a592-15df64a66ecf-kube-api-access-x6zs6\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc\" (UID: \"1da9c52e-d938-4ead-a592-15df64a66ecf\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc"
Sep 08 13:39:32.227832 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:32.227804    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1da9c52e-d938-4ead-a592-15df64a66ecf-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc\" (UID: \"1da9c52e-d938-4ead-a592-15df64a66ecf\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc"
Sep 08 13:39:32.235819 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:32.235791    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6zs6\" (UniqueName: \"kubernetes.io/projected/1da9c52e-d938-4ead-a592-15df64a66ecf-kube-api-access-x6zs6\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc\" (UID: \"1da9c52e-d938-4ead-a592-15df64a66ecf\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc"
Sep 08 13:39:32.318142 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:32.318042    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc"
Sep 08 13:39:32.616006 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:32.615928    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc"]
Sep 08 13:39:32.620118 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:39:32.620088    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da9c52e_d938_4ead_a592_15df64a66ecf.slice/crio-a4349ff5483989a3ab9cfd0a9b7cb4e979186b136c1a2ff9e430b51c07e81468 WatchSource:0}: Error finding container a4349ff5483989a3ab9cfd0a9b7cb4e979186b136c1a2ff9e430b51c07e81468: Status 404 returned error can't find the container with id a4349ff5483989a3ab9cfd0a9b7cb4e979186b136c1a2ff9e430b51c07e81468
Sep 08 13:39:32.922318 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:32.922205    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc" event={"ID":"1da9c52e-d938-4ead-a592-15df64a66ecf","Type":"ContainerStarted","Data":"72581672553f48525019c1def646de6a1243cc37e4911ddc20eca310737fc26a"}
Sep 08 13:39:32.922318 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:32.922263    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc" event={"ID":"1da9c52e-d938-4ead-a592-15df64a66ecf","Type":"ContainerStarted","Data":"a4349ff5483989a3ab9cfd0a9b7cb4e979186b136c1a2ff9e430b51c07e81468"}
Sep 08 13:39:32.922525 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:32.922351    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc"
Sep 08 13:39:32.955420 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:39:32.955341    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc" podStartSLOduration=1.9553177160000001 podStartE2EDuration="1.955317716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:39:32.951866794 +0000 UTC m=+1404.412511607" watchObservedRunningTime="2026-09-08 13:39:32.955317716 +0000 UTC m=+1404.415962529"
Sep 08 13:40:03.929375 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:03.929282    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc"
Sep 08 13:40:05.726411 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:05.726359    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc"]
Sep 08 13:40:13.930191 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:13.930149    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc" podUID="1da9c52e-d938-4ead-a592-15df64a66ecf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:40:15.913570 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:15.913489    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5"]
Sep 08 13:40:16.502898 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:16.502799    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5" podUID="27cf4e93-f79b-4a82-9446-23f8e5afebc4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:40:20.753074 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:20.753030    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc" podUID="1da9c52e-d938-4ead-a592-15df64a66ecf" containerName="main" containerID="cri-o://72581672553f48525019c1def646de6a1243cc37e4911ddc20eca310737fc26a" gracePeriod=45
Sep 08 13:40:23.929380 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:23.929337    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc" podUID="1da9c52e-d938-4ead-a592-15df64a66ecf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:40:26.502377 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:26.502338    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5" podUID="27cf4e93-f79b-4a82-9446-23f8e5afebc4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:40:30.934662 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:30.934614    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5" podUID="27cf4e93-f79b-4a82-9446-23f8e5afebc4" containerName="main" containerID="cri-o://d72b7bc6276e52e9d81c61e84a293b78bc7f31a6a62d35abc9349ca223c3b253" gracePeriod=45
Sep 08 13:40:33.929194 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:33.929151    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc" podUID="1da9c52e-d938-4ead-a592-15df64a66ecf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:40:33.929677 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:33.929262    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc"
Sep 08 13:40:36.503350 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:36.503310    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5" podUID="27cf4e93-f79b-4a82-9446-23f8e5afebc4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:40:36.503739 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:36.503419    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5"
Sep 08 13:40:43.929419 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:43.929375    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc" podUID="1da9c52e-d938-4ead-a592-15df64a66ecf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:40:46.502435 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:46.502395    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5" podUID="27cf4e93-f79b-4a82-9446-23f8e5afebc4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:40:50.999693 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:50.999666    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc"
Sep 08 13:40:51.125228 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:51.125131    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6zs6\" (UniqueName: \"kubernetes.io/projected/1da9c52e-d938-4ead-a592-15df64a66ecf-kube-api-access-x6zs6\") pod \"1da9c52e-d938-4ead-a592-15df64a66ecf\" (UID: \"1da9c52e-d938-4ead-a592-15df64a66ecf\") "
Sep 08 13:40:51.125228 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:51.125214    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1da9c52e-d938-4ead-a592-15df64a66ecf-tls-certs\") pod \"1da9c52e-d938-4ead-a592-15df64a66ecf\" (UID: \"1da9c52e-d938-4ead-a592-15df64a66ecf\") "
Sep 08 13:40:51.127435 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:51.127403    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1da9c52e-d938-4ead-a592-15df64a66ecf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1da9c52e-d938-4ead-a592-15df64a66ecf" (UID: "1da9c52e-d938-4ead-a592-15df64a66ecf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:40:51.127550 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:51.127451    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da9c52e-d938-4ead-a592-15df64a66ecf-kube-api-access-x6zs6" (OuterVolumeSpecName: "kube-api-access-x6zs6") pod "1da9c52e-d938-4ead-a592-15df64a66ecf" (UID: "1da9c52e-d938-4ead-a592-15df64a66ecf"). InnerVolumeSpecName "kube-api-access-x6zs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:40:51.226851 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:51.226815    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6zs6\" (UniqueName: \"kubernetes.io/projected/1da9c52e-d938-4ead-a592-15df64a66ecf-kube-api-access-x6zs6\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:40:51.226851 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:51.226846    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1da9c52e-d938-4ead-a592-15df64a66ecf-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:40:51.365649 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:51.365611    2561 generic.go:356] "Generic (PLEG): container finished" podID="1da9c52e-d938-4ead-a592-15df64a66ecf" containerID="72581672553f48525019c1def646de6a1243cc37e4911ddc20eca310737fc26a" exitCode=0
Sep 08 13:40:51.365829 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:51.365656    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc" event={"ID":"1da9c52e-d938-4ead-a592-15df64a66ecf","Type":"ContainerDied","Data":"72581672553f48525019c1def646de6a1243cc37e4911ddc20eca310737fc26a"}
Sep 08 13:40:51.365829 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:51.365695    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc" event={"ID":"1da9c52e-d938-4ead-a592-15df64a66ecf","Type":"ContainerDied","Data":"a4349ff5483989a3ab9cfd0a9b7cb4e979186b136c1a2ff9e430b51c07e81468"}
Sep 08 13:40:51.365829 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:51.365718    2561 scope.go:117] "RemoveContainer" containerID="72581672553f48525019c1def646de6a1243cc37e4911ddc20eca310737fc26a"
Sep 08 13:40:51.365829 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:51.365657    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc"
Sep 08 13:40:51.375604 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:51.375584    2561 scope.go:117] "RemoveContainer" containerID="72581672553f48525019c1def646de6a1243cc37e4911ddc20eca310737fc26a"
Sep 08 13:40:51.375909 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:40:51.375886    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72581672553f48525019c1def646de6a1243cc37e4911ddc20eca310737fc26a\": container with ID starting with 72581672553f48525019c1def646de6a1243cc37e4911ddc20eca310737fc26a not found: ID does not exist" containerID="72581672553f48525019c1def646de6a1243cc37e4911ddc20eca310737fc26a"
Sep 08 13:40:51.375959 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:51.375917    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72581672553f48525019c1def646de6a1243cc37e4911ddc20eca310737fc26a"} err="failed to get container status \"72581672553f48525019c1def646de6a1243cc37e4911ddc20eca310737fc26a\": rpc error: code = NotFound desc = could not find container \"72581672553f48525019c1def646de6a1243cc37e4911ddc20eca310737fc26a\": container with ID starting with 72581672553f48525019c1def646de6a1243cc37e4911ddc20eca310737fc26a not found: ID does not exist"
Sep 08 13:40:51.394397 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:51.394367    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc"]
Sep 08 13:40:51.397840 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:51.397813    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9w6dpc"]
Sep 08 13:40:53.006936 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:53.006901    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1da9c52e-d938-4ead-a592-15df64a66ecf" path="/var/lib/kubelet/pods/1da9c52e-d938-4ead-a592-15df64a66ecf/volumes"
Sep 08 13:40:56.502938 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:40:56.502891    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5" podUID="27cf4e93-f79b-4a82-9446-23f8e5afebc4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:41:01.189024 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.188996    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5"
Sep 08 13:41:01.320311 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.320184    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnlzr\" (UniqueName: \"kubernetes.io/projected/27cf4e93-f79b-4a82-9446-23f8e5afebc4-kube-api-access-nnlzr\") pod \"27cf4e93-f79b-4a82-9446-23f8e5afebc4\" (UID: \"27cf4e93-f79b-4a82-9446-23f8e5afebc4\") "
Sep 08 13:41:01.320311 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.320265    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27cf4e93-f79b-4a82-9446-23f8e5afebc4-tls-certs\") pod \"27cf4e93-f79b-4a82-9446-23f8e5afebc4\" (UID: \"27cf4e93-f79b-4a82-9446-23f8e5afebc4\") "
Sep 08 13:41:01.322634 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.322607    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27cf4e93-f79b-4a82-9446-23f8e5afebc4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27cf4e93-f79b-4a82-9446-23f8e5afebc4" (UID: "27cf4e93-f79b-4a82-9446-23f8e5afebc4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:41:01.322750 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.322641    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27cf4e93-f79b-4a82-9446-23f8e5afebc4-kube-api-access-nnlzr" (OuterVolumeSpecName: "kube-api-access-nnlzr") pod "27cf4e93-f79b-4a82-9446-23f8e5afebc4" (UID: "27cf4e93-f79b-4a82-9446-23f8e5afebc4"). InnerVolumeSpecName "kube-api-access-nnlzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:41:01.419776 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.419741    2561 generic.go:356] "Generic (PLEG): container finished" podID="27cf4e93-f79b-4a82-9446-23f8e5afebc4" containerID="d72b7bc6276e52e9d81c61e84a293b78bc7f31a6a62d35abc9349ca223c3b253" exitCode=0
Sep 08 13:41:01.419953 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.419793    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5"
Sep 08 13:41:01.419953 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.419799    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5" event={"ID":"27cf4e93-f79b-4a82-9446-23f8e5afebc4","Type":"ContainerDied","Data":"d72b7bc6276e52e9d81c61e84a293b78bc7f31a6a62d35abc9349ca223c3b253"}
Sep 08 13:41:01.419953 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.419855    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5" event={"ID":"27cf4e93-f79b-4a82-9446-23f8e5afebc4","Type":"ContainerDied","Data":"f32b28be6cd961410f7d5ad973b3b7b5975306e753912ad87d4bb2180418d80b"}
Sep 08 13:41:01.419953 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.419884    2561 scope.go:117] "RemoveContainer" containerID="d72b7bc6276e52e9d81c61e84a293b78bc7f31a6a62d35abc9349ca223c3b253"
Sep 08 13:41:01.420890 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.420865    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27cf4e93-f79b-4a82-9446-23f8e5afebc4-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:41:01.420890 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.420891    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnlzr\" (UniqueName: \"kubernetes.io/projected/27cf4e93-f79b-4a82-9446-23f8e5afebc4-kube-api-access-nnlzr\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:41:01.429901 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.429879    2561 scope.go:117] "RemoveContainer" containerID="d72b7bc6276e52e9d81c61e84a293b78bc7f31a6a62d35abc9349ca223c3b253"
Sep 08 13:41:01.430209 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:41:01.430187    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d72b7bc6276e52e9d81c61e84a293b78bc7f31a6a62d35abc9349ca223c3b253\": container with ID starting with d72b7bc6276e52e9d81c61e84a293b78bc7f31a6a62d35abc9349ca223c3b253 not found: ID does not exist" containerID="d72b7bc6276e52e9d81c61e84a293b78bc7f31a6a62d35abc9349ca223c3b253"
Sep 08 13:41:01.430278 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.430217    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d72b7bc6276e52e9d81c61e84a293b78bc7f31a6a62d35abc9349ca223c3b253"} err="failed to get container status \"d72b7bc6276e52e9d81c61e84a293b78bc7f31a6a62d35abc9349ca223c3b253\": rpc error: code = NotFound desc = could not find container \"d72b7bc6276e52e9d81c61e84a293b78bc7f31a6a62d35abc9349ca223c3b253\": container with ID starting with d72b7bc6276e52e9d81c61e84a293b78bc7f31a6a62d35abc9349ca223c3b253 not found: ID does not exist"
Sep 08 13:41:01.452610 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.452570    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5"]
Sep 08 13:41:01.465448 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.465417    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crbxl5"]
Sep 08 13:41:01.938835 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.938800    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl"]
Sep 08 13:41:01.939157 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.939145    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1da9c52e-d938-4ead-a592-15df64a66ecf" containerName="main"
Sep 08 13:41:01.939202 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.939159    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1da9c52e-d938-4ead-a592-15df64a66ecf" containerName="main"
Sep 08 13:41:01.939202 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.939172    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27cf4e93-f79b-4a82-9446-23f8e5afebc4" containerName="main"
Sep 08 13:41:01.939202 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.939178    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="27cf4e93-f79b-4a82-9446-23f8e5afebc4" containerName="main"
Sep 08 13:41:01.939312 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.939227    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1da9c52e-d938-4ead-a592-15df64a66ecf" containerName="main"
Sep 08 13:41:01.939312 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.939237    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="27cf4e93-f79b-4a82-9446-23f8e5afebc4" containerName="main"
Sep 08 13:41:01.943532 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.943511    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl"
Sep 08 13:41:01.952260 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.952229    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-v7lbz\""
Sep 08 13:41:01.952428 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.952268    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 08 13:41:01.952428 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.952306    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-z8587\""
Sep 08 13:41:01.952428 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.952357    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 08 13:41:01.953368 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.953350    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 08 13:41:01.971204 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:01.971171    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl"]
Sep 08 13:41:02.026343 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:02.026305    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq4j5\" (UniqueName: \"kubernetes.io/projected/6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab-kube-api-access-pq4j5\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl\" (UID: \"6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl"
Sep 08 13:41:02.026540 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:02.026391    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl\" (UID: \"6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl"
Sep 08 13:41:02.127341 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:02.127284    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl\" (UID: \"6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl"
Sep 08 13:41:02.127511 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:02.127438    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq4j5\" (UniqueName: \"kubernetes.io/projected/6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab-kube-api-access-pq4j5\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl\" (UID: \"6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl"
Sep 08 13:41:02.129730 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:02.129704    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl\" (UID: \"6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl"
Sep 08 13:41:02.139593 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:02.139571    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq4j5\" (UniqueName: \"kubernetes.io/projected/6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab-kube-api-access-pq4j5\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl\" (UID: \"6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl"
Sep 08 13:41:02.255622 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:02.255533    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl"
Sep 08 13:41:02.544751 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:02.544661    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl"]
Sep 08 13:41:02.551457 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:41:02.551425    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5ae8c4_c1bd_4338_a8d9_023fdc6056ab.slice/crio-b943cfca3a5aaf2de4802523e6a386c3d69a69a078ab7adad99760f58e8e05a8 WatchSource:0}: Error finding container b943cfca3a5aaf2de4802523e6a386c3d69a69a078ab7adad99760f58e8e05a8: Status 404 returned error can't find the container with id b943cfca3a5aaf2de4802523e6a386c3d69a69a078ab7adad99760f58e8e05a8
Sep 08 13:41:03.013966 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:03.013930    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27cf4e93-f79b-4a82-9446-23f8e5afebc4" path="/var/lib/kubelet/pods/27cf4e93-f79b-4a82-9446-23f8e5afebc4/volumes"
Sep 08 13:41:03.432539 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:03.432432    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl" event={"ID":"6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab","Type":"ContainerStarted","Data":"a09a3e24d596d8f4e6e88d01780afb0bfd27f7e2f420c58ea57dc166b2bc2f27"}
Sep 08 13:41:03.432539 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:03.432496    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl" event={"ID":"6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab","Type":"ContainerStarted","Data":"b943cfca3a5aaf2de4802523e6a386c3d69a69a078ab7adad99760f58e8e05a8"}
Sep 08 13:41:03.432539 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:03.432526    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl"
Sep 08 13:41:03.459787 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:03.459711    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl" podStartSLOduration=2.459685699 podStartE2EDuration="2.459685699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:41:03.456174801 +0000 UTC m=+1494.916819611" watchObservedRunningTime="2026-09-08 13:41:03.459685699 +0000 UTC m=+1494.920330510"
Sep 08 13:41:09.059730 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:09.059700    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 13:41:09.061230 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:09.061207    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 13:41:24.928902 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:24.928866    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk"]
Sep 08 13:41:24.933527 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:24.933499    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk"
Sep 08 13:41:24.936434 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:24.936412    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-jnbwc\""
Sep 08 13:41:24.937003 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:24.936982    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-tdbxb\""
Sep 08 13:41:24.937347 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:24.937326    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 08 13:41:24.937420 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:24.937366    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 08 13:41:24.937478 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:24.937419    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 08 13:41:24.956080 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:24.956049    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk"]
Sep 08 13:41:25.034716 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:25.034674    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a64d1340-0d6a-44e3-ae37-bd48e7cd1565-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk\" (UID: \"a64d1340-0d6a-44e3-ae37-bd48e7cd1565\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk"
Sep 08 13:41:25.034893 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:25.034782    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzhgl\" (UniqueName: \"kubernetes.io/projected/a64d1340-0d6a-44e3-ae37-bd48e7cd1565-kube-api-access-lzhgl\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk\" (UID: \"a64d1340-0d6a-44e3-ae37-bd48e7cd1565\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk"
Sep 08 13:41:25.135723 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:25.135681    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzhgl\" (UniqueName: \"kubernetes.io/projected/a64d1340-0d6a-44e3-ae37-bd48e7cd1565-kube-api-access-lzhgl\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk\" (UID: \"a64d1340-0d6a-44e3-ae37-bd48e7cd1565\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk"
Sep 08 13:41:25.135908 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:25.135772    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a64d1340-0d6a-44e3-ae37-bd48e7cd1565-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk\" (UID: \"a64d1340-0d6a-44e3-ae37-bd48e7cd1565\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk"
Sep 08 13:41:25.138321 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:25.138279    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a64d1340-0d6a-44e3-ae37-bd48e7cd1565-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk\" (UID: \"a64d1340-0d6a-44e3-ae37-bd48e7cd1565\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk"
Sep 08 13:41:25.145599 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:25.145573    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzhgl\" (UniqueName: \"kubernetes.io/projected/a64d1340-0d6a-44e3-ae37-bd48e7cd1565-kube-api-access-lzhgl\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk\" (UID: \"a64d1340-0d6a-44e3-ae37-bd48e7cd1565\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk"
Sep 08 13:41:25.245457 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:25.245414    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk"
Sep 08 13:41:25.528859 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:25.528834    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk"]
Sep 08 13:41:25.531599 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:41:25.531573    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64d1340_0d6a_44e3_ae37_bd48e7cd1565.slice/crio-c5d02f8dc602cc71a14bf13e007efd3d552ebee84c803a63d64f9db225fab872 WatchSource:0}: Error finding container c5d02f8dc602cc71a14bf13e007efd3d552ebee84c803a63d64f9db225fab872: Status 404 returned error can't find the container with id c5d02f8dc602cc71a14bf13e007efd3d552ebee84c803a63d64f9db225fab872
Sep 08 13:41:25.557827 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:25.557797    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk" event={"ID":"a64d1340-0d6a-44e3-ae37-bd48e7cd1565","Type":"ContainerStarted","Data":"c5d02f8dc602cc71a14bf13e007efd3d552ebee84c803a63d64f9db225fab872"}
Sep 08 13:41:26.565506 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:26.565467    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk" event={"ID":"a64d1340-0d6a-44e3-ae37-bd48e7cd1565","Type":"ContainerStarted","Data":"0be2160679ac10c1425613fad62891b0b575190429e932e4145577e4eb17f4af"}
Sep 08 13:41:26.565998 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:26.565612    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk"
Sep 08 13:41:34.439620 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:34.439540    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl"
Sep 08 13:41:34.477199 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:34.477146    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk" podStartSLOduration=10.477130653 podStartE2EDuration="10.477130653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:41:26.589799005 +0000 UTC m=+1518.050443815" watchObservedRunningTime="2026-09-08 13:41:34.477130653 +0000 UTC m=+1525.937775445"
Sep 08 13:41:38.374618 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:38.374581    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl"]
Sep 08 13:41:44.440001 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:44.439967    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl" podUID="6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:41:48.985789 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:48.985748    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 08 13:41:48.985789 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:41:48.985786    2561 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl" podUID="6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab" containerName="main" containerID="cri-o://a09a3e24d596d8f4e6e88d01780afb0bfd27f7e2f420c58ea57dc166b2bc2f27"
Sep 08 13:41:48.986188 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:48.985830    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl" podUID="6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab" containerName="main" containerID="cri-o://a09a3e24d596d8f4e6e88d01780afb0bfd27f7e2f420c58ea57dc166b2bc2f27" gracePeriod=50
Sep 08 13:41:49.112908 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:49.112882    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl_6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab/main/0.log"
Sep 08 13:41:49.115354 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:49.115333    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl"
Sep 08 13:41:49.257824 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:49.257740    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab-tls-certs\") pod \"6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab\" (UID: \"6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab\") "
Sep 08 13:41:49.257824 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:49.257780    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pq4j5\" (UniqueName: \"kubernetes.io/projected/6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab-kube-api-access-pq4j5\") pod \"6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab\" (UID: \"6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab\") "
Sep 08 13:41:49.260043 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:49.260012    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab-kube-api-access-pq4j5" (OuterVolumeSpecName: "kube-api-access-pq4j5") pod "6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab" (UID: "6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab"). InnerVolumeSpecName "kube-api-access-pq4j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:41:49.260138 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:49.260015    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab" (UID: "6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:41:49.358587 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:49.358549    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:41:49.358587 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:49.358580    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq4j5\" (UniqueName: \"kubernetes.io/projected/6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab-kube-api-access-pq4j5\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:41:49.689937 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:49.689907    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl_6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab/main/0.log"
Sep 08 13:41:49.690135 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:49.689957    2561 generic.go:356] "Generic (PLEG): container finished" podID="6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab" containerID="a09a3e24d596d8f4e6e88d01780afb0bfd27f7e2f420c58ea57dc166b2bc2f27" exitCode=1
Sep 08 13:41:49.690135 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:49.690017    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl"
Sep 08 13:41:49.690135 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:49.690047    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl" event={"ID":"6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab","Type":"ContainerDied","Data":"a09a3e24d596d8f4e6e88d01780afb0bfd27f7e2f420c58ea57dc166b2bc2f27"}
Sep 08 13:41:49.690135 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:49.690116    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl" event={"ID":"6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab","Type":"ContainerDied","Data":"b943cfca3a5aaf2de4802523e6a386c3d69a69a078ab7adad99760f58e8e05a8"}
Sep 08 13:41:49.690334 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:49.690140    2561 scope.go:117] "RemoveContainer" containerID="a09a3e24d596d8f4e6e88d01780afb0bfd27f7e2f420c58ea57dc166b2bc2f27"
Sep 08 13:41:49.699423 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:49.699402    2561 scope.go:117] "RemoveContainer" containerID="a09a3e24d596d8f4e6e88d01780afb0bfd27f7e2f420c58ea57dc166b2bc2f27"
Sep 08 13:41:49.699721 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:41:49.699703    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a09a3e24d596d8f4e6e88d01780afb0bfd27f7e2f420c58ea57dc166b2bc2f27\": container with ID starting with a09a3e24d596d8f4e6e88d01780afb0bfd27f7e2f420c58ea57dc166b2bc2f27 not found: ID does not exist" containerID="a09a3e24d596d8f4e6e88d01780afb0bfd27f7e2f420c58ea57dc166b2bc2f27"
Sep 08 13:41:49.699772 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:49.699729    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a09a3e24d596d8f4e6e88d01780afb0bfd27f7e2f420c58ea57dc166b2bc2f27"} err="failed to get container status \"a09a3e24d596d8f4e6e88d01780afb0bfd27f7e2f420c58ea57dc166b2bc2f27\": rpc error: code = NotFound desc = could not find container \"a09a3e24d596d8f4e6e88d01780afb0bfd27f7e2f420c58ea57dc166b2bc2f27\": container with ID starting with a09a3e24d596d8f4e6e88d01780afb0bfd27f7e2f420c58ea57dc166b2bc2f27 not found: ID does not exist"
Sep 08 13:41:49.720611 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:49.720555    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl"]
Sep 08 13:41:49.722887 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:49.722864    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5b7xxl"]
Sep 08 13:41:51.006752 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:51.006715    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab" path="/var/lib/kubelet/pods/6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab/volumes"
Sep 08 13:41:57.572613 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:41:57.572583    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk"
Sep 08 13:43:29.946602 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:43:29.946566    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk"]
Sep 08 13:43:37.572411 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:43:37.572372    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk" podUID="a64d1340-0d6a-44e3-ae37-bd48e7cd1565" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:43:44.967084 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:43:44.967020    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk" podUID="a64d1340-0d6a-44e3-ae37-bd48e7cd1565" containerName="main" containerID="cri-o://0be2160679ac10c1425613fad62891b0b575190429e932e4145577e4eb17f4af" gracePeriod=45
Sep 08 13:43:47.572617 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:43:47.572577    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk" podUID="a64d1340-0d6a-44e3-ae37-bd48e7cd1565" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:43:57.573035 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:43:57.572999    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk" podUID="a64d1340-0d6a-44e3-ae37-bd48e7cd1565" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:43:57.573433 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:43:57.573091    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk"
Sep 08 13:44:07.572778 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:07.572741    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk" podUID="a64d1340-0d6a-44e3-ae37-bd48e7cd1565" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:44:15.225563 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:15.225501    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk"
Sep 08 13:44:15.298779 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:15.298745    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a64d1340-0d6a-44e3-ae37-bd48e7cd1565-tls-certs\") pod \"a64d1340-0d6a-44e3-ae37-bd48e7cd1565\" (UID: \"a64d1340-0d6a-44e3-ae37-bd48e7cd1565\") "
Sep 08 13:44:15.298946 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:15.298835    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzhgl\" (UniqueName: \"kubernetes.io/projected/a64d1340-0d6a-44e3-ae37-bd48e7cd1565-kube-api-access-lzhgl\") pod \"a64d1340-0d6a-44e3-ae37-bd48e7cd1565\" (UID: \"a64d1340-0d6a-44e3-ae37-bd48e7cd1565\") "
Sep 08 13:44:15.301107 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:15.301071    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64d1340-0d6a-44e3-ae37-bd48e7cd1565-kube-api-access-lzhgl" (OuterVolumeSpecName: "kube-api-access-lzhgl") pod "a64d1340-0d6a-44e3-ae37-bd48e7cd1565" (UID: "a64d1340-0d6a-44e3-ae37-bd48e7cd1565"). InnerVolumeSpecName "kube-api-access-lzhgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:44:15.301107 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:15.301068    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a64d1340-0d6a-44e3-ae37-bd48e7cd1565-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a64d1340-0d6a-44e3-ae37-bd48e7cd1565" (UID: "a64d1340-0d6a-44e3-ae37-bd48e7cd1565"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:44:15.399911 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:15.399867    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzhgl\" (UniqueName: \"kubernetes.io/projected/a64d1340-0d6a-44e3-ae37-bd48e7cd1565-kube-api-access-lzhgl\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:44:15.399911 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:15.399903    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a64d1340-0d6a-44e3-ae37-bd48e7cd1565-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:44:15.471682 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:15.471643    2561 generic.go:356] "Generic (PLEG): container finished" podID="a64d1340-0d6a-44e3-ae37-bd48e7cd1565" containerID="0be2160679ac10c1425613fad62891b0b575190429e932e4145577e4eb17f4af" exitCode=0
Sep 08 13:44:15.471852 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:15.471690    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk"
Sep 08 13:44:15.471852 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:15.471724    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk" event={"ID":"a64d1340-0d6a-44e3-ae37-bd48e7cd1565","Type":"ContainerDied","Data":"0be2160679ac10c1425613fad62891b0b575190429e932e4145577e4eb17f4af"}
Sep 08 13:44:15.471852 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:15.471780    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk" event={"ID":"a64d1340-0d6a-44e3-ae37-bd48e7cd1565","Type":"ContainerDied","Data":"c5d02f8dc602cc71a14bf13e007efd3d552ebee84c803a63d64f9db225fab872"}
Sep 08 13:44:15.471852 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:15.471804    2561 scope.go:117] "RemoveContainer" containerID="0be2160679ac10c1425613fad62891b0b575190429e932e4145577e4eb17f4af"
Sep 08 13:44:15.483151 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:15.483134    2561 scope.go:117] "RemoveContainer" containerID="0be2160679ac10c1425613fad62891b0b575190429e932e4145577e4eb17f4af"
Sep 08 13:44:15.483441 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:44:15.483422    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0be2160679ac10c1425613fad62891b0b575190429e932e4145577e4eb17f4af\": container with ID starting with 0be2160679ac10c1425613fad62891b0b575190429e932e4145577e4eb17f4af not found: ID does not exist" containerID="0be2160679ac10c1425613fad62891b0b575190429e932e4145577e4eb17f4af"
Sep 08 13:44:15.483499 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:15.483449    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0be2160679ac10c1425613fad62891b0b575190429e932e4145577e4eb17f4af"} err="failed to get container status \"0be2160679ac10c1425613fad62891b0b575190429e932e4145577e4eb17f4af\": rpc error: code = NotFound desc = could not find container \"0be2160679ac10c1425613fad62891b0b575190429e932e4145577e4eb17f4af\": container with ID starting with 0be2160679ac10c1425613fad62891b0b575190429e932e4145577e4eb17f4af not found: ID does not exist"
Sep 08 13:44:15.494814 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:15.494789    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk"]
Sep 08 13:44:15.498484 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:15.498465    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rbbgk"]
Sep 08 13:44:17.006742 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:17.006708    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a64d1340-0d6a-44e3-ae37-bd48e7cd1565" path="/var/lib/kubelet/pods/a64d1340-0d6a-44e3-ae37-bd48e7cd1565/volumes"
Sep 08 13:44:25.603218 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.603178    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"]
Sep 08 13:44:25.603728 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.603710    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab" containerName="main"
Sep 08 13:44:25.603786 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.603733    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab" containerName="main"
Sep 08 13:44:25.603786 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.603750    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a64d1340-0d6a-44e3-ae37-bd48e7cd1565" containerName="main"
Sep 08 13:44:25.603786 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.603759    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a64d1340-0d6a-44e3-ae37-bd48e7cd1565" containerName="main"
Sep 08 13:44:25.603886 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.603869    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a64d1340-0d6a-44e3-ae37-bd48e7cd1565" containerName="main"
Sep 08 13:44:25.603920 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.603890    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f5ae8c4-c1bd-4338-a8d9-023fdc6056ab" containerName="main"
Sep 08 13:44:25.608219 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.608198    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:25.611235 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.611208    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 08 13:44:25.611380 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.611271    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 08 13:44:25.611380 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.611315    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-9db4b\""
Sep 08 13:44:25.611380 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.611331    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v1-kserve-self-signed-certs\""
Sep 08 13:44:25.620393 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.620367    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"]
Sep 08 13:44:25.693019 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.692981    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-v99b4\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:25.693185 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.693042    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4928\" (UniqueName: \"kubernetes.io/projected/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-kube-api-access-c4928\") pod \"canary-v1-kserve-65d58ccc9-v99b4\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:25.693185 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.693152    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-home\") pod \"canary-v1-kserve-65d58ccc9-v99b4\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:25.693185 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.693179    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-model-cache\") pod \"canary-v1-kserve-65d58ccc9-v99b4\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:25.693316 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.693207    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-v99b4\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:25.693316 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.693271    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-dshm\") pod \"canary-v1-kserve-65d58ccc9-v99b4\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:25.794536 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.794495    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-home\") pod \"canary-v1-kserve-65d58ccc9-v99b4\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:25.794536 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.794532    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-model-cache\") pod \"canary-v1-kserve-65d58ccc9-v99b4\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:25.794784 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.794556    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-v99b4\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:25.794784 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.794584    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-dshm\") pod \"canary-v1-kserve-65d58ccc9-v99b4\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:25.794784 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.794616    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-v99b4\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:25.794784 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.794678    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4928\" (UniqueName: \"kubernetes.io/projected/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-kube-api-access-c4928\") pod \"canary-v1-kserve-65d58ccc9-v99b4\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:25.795009 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.794979    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-home\") pod \"canary-v1-kserve-65d58ccc9-v99b4\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:25.795075 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.795020    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-model-cache\") pod \"canary-v1-kserve-65d58ccc9-v99b4\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:25.795137 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.795075    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-v99b4\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:25.796918 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.796889    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-dshm\") pod \"canary-v1-kserve-65d58ccc9-v99b4\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:25.797155 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.797136    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-v99b4\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:25.803684 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.803660    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4928\" (UniqueName: \"kubernetes.io/projected/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-kube-api-access-c4928\") pod \"canary-v1-kserve-65d58ccc9-v99b4\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:25.924553 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:25.924464    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:26.199866 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:26.199841    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"]
Sep 08 13:44:26.202126 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:44:26.202093    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcecd6812_086a_4788_a9d6_e0ad3bc83ddc.slice/crio-9cd532b3011fc404f27599875a0a010fe5bf8e66e63f05f7d154223d88e84ce4 WatchSource:0}: Error finding container 9cd532b3011fc404f27599875a0a010fe5bf8e66e63f05f7d154223d88e84ce4: Status 404 returned error can't find the container with id 9cd532b3011fc404f27599875a0a010fe5bf8e66e63f05f7d154223d88e84ce4
Sep 08 13:44:26.203899 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:26.203883    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:44:26.530900 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:26.530800    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4" event={"ID":"cecd6812-086a-4788-a9d6-e0ad3bc83ddc","Type":"ContainerStarted","Data":"9cd532b3011fc404f27599875a0a010fe5bf8e66e63f05f7d154223d88e84ce4"}
Sep 08 13:44:27.541144 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:27.540738    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4" event={"ID":"cecd6812-086a-4788-a9d6-e0ad3bc83ddc","Type":"ContainerStarted","Data":"2564777e05a881938f6d2e6c5197fac68c2581931810fa40834c61847c2b8a6c"}
Sep 08 13:44:27.583110 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:27.583058    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4" podStartSLOduration=1.594525497 podStartE2EDuration="2.583040773s" totalImagesPullingTime="988.515276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:44:27.580179439 +0000 UTC m=+1699.040824262" watchObservedRunningTime="2026-09-08 13:44:27.583040773 +0000 UTC m=+1699.043685642"
Sep 08 13:44:27.925269 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:27.925192    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:33.934218 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:33.934143    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"]
Sep 08 13:44:33.938026 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:33.938004    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:33.940778 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:33.940734    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v2-kserve-self-signed-certs\""
Sep 08 13:44:33.952430 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:33.952397    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"]
Sep 08 13:44:34.070098 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:34.070052    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-dshm\") pod \"canary-v2-kserve-7f65dbbccc-7rfjh\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:34.070098 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:34.070105    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-7rfjh\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:34.070360 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:34.070128    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-7rfjh\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:34.070360 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:34.070212    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-home\") pod \"canary-v2-kserve-7f65dbbccc-7rfjh\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:34.070360 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:34.070338    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-7rfjh\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:34.070478 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:34.070401    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bn88\" (UniqueName: \"kubernetes.io/projected/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-kube-api-access-9bn88\") pod \"canary-v2-kserve-7f65dbbccc-7rfjh\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:34.171244 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:34.171204    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-7rfjh\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:34.171481 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:34.171275    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bn88\" (UniqueName: \"kubernetes.io/projected/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-kube-api-access-9bn88\") pod \"canary-v2-kserve-7f65dbbccc-7rfjh\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:34.171481 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:34.171338    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-dshm\") pod \"canary-v2-kserve-7f65dbbccc-7rfjh\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:34.171481 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:34.171367    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-7rfjh\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:34.171481 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:34.171408    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-7rfjh\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:34.171481 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:34.171440    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-home\") pod \"canary-v2-kserve-7f65dbbccc-7rfjh\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:34.171879 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:34.171855    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-home\") pod \"canary-v2-kserve-7f65dbbccc-7rfjh\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:34.172011 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:34.171897    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-7rfjh\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:34.172179 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:34.172157    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-7rfjh\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:34.173812 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:34.173785    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-dshm\") pod \"canary-v2-kserve-7f65dbbccc-7rfjh\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:34.174146 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:34.174127    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-7rfjh\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:34.184119 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:34.184097    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bn88\" (UniqueName: \"kubernetes.io/projected/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-kube-api-access-9bn88\") pod \"canary-v2-kserve-7f65dbbccc-7rfjh\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:34.250809 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:34.250714    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:34.535500 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:34.535407    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"]
Sep 08 13:44:34.544112 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:44:34.544078    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1aeb9c_8b21_4ff3_b0c0_b7581eb13714.slice/crio-2cc3cb21f724f7b796faa452445cb0c9caf60acd9d30b3f9c273dbe6ae3a91a3 WatchSource:0}: Error finding container 2cc3cb21f724f7b796faa452445cb0c9caf60acd9d30b3f9c273dbe6ae3a91a3: Status 404 returned error can't find the container with id 2cc3cb21f724f7b796faa452445cb0c9caf60acd9d30b3f9c273dbe6ae3a91a3
Sep 08 13:44:34.579602 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:34.579573    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh" event={"ID":"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714","Type":"ContainerStarted","Data":"2cc3cb21f724f7b796faa452445cb0c9caf60acd9d30b3f9c273dbe6ae3a91a3"}
Sep 08 13:44:35.587767 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:35.587730    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh" event={"ID":"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714","Type":"ContainerStarted","Data":"12489094cc9941f489b5c1cebd5ab3bdbc0f77d6651ea97591ff665d046f3546"}
Sep 08 13:44:35.612273 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:35.612220    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh" podStartSLOduration=2.612199322 podStartE2EDuration="2.612199322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:44:35.610482123 +0000 UTC m=+1707.071126957" watchObservedRunningTime="2026-09-08 13:44:35.612199322 +0000 UTC m=+1707.072844131"
Sep 08 13:44:35.924795 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:35.924705    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:35.937573 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:35.937542    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:36.251208 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:36.251168    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:36.605298 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:36.605213    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:44:44.251913 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:44.251870    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:44.264620 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:44.264593    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:44:44.652774 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:44:44.652690    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:46:02.549579 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:02.549505    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj"]
Sep 08 13:46:02.553011 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:02.552988    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj"
Sep 08 13:46:02.561601 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:02.561575    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 08 13:46:02.561715 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:02.561612    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-t8rsf\""
Sep 08 13:46:02.561715 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:02.561642    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 08 13:46:02.561715 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:02.561576    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 08 13:46:02.561715 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:02.561698    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-qpg7m\""
Sep 08 13:46:02.570022 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:02.569992    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj"]
Sep 08 13:46:02.681623 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:02.681585    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4xzq\" (UniqueName: \"kubernetes.io/projected/8535bf4a-f163-47e2-8baa-e4d2658480ee-kube-api-access-g4xzq\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj\" (UID: \"8535bf4a-f163-47e2-8baa-e4d2658480ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj"
Sep 08 13:46:02.681816 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:02.681666    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8535bf4a-f163-47e2-8baa-e4d2658480ee-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj\" (UID: \"8535bf4a-f163-47e2-8baa-e4d2658480ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj"
Sep 08 13:46:02.782462 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:02.782423    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4xzq\" (UniqueName: \"kubernetes.io/projected/8535bf4a-f163-47e2-8baa-e4d2658480ee-kube-api-access-g4xzq\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj\" (UID: \"8535bf4a-f163-47e2-8baa-e4d2658480ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj"
Sep 08 13:46:02.782636 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:02.782496    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8535bf4a-f163-47e2-8baa-e4d2658480ee-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj\" (UID: \"8535bf4a-f163-47e2-8baa-e4d2658480ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj"
Sep 08 13:46:02.785158 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:02.785126    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8535bf4a-f163-47e2-8baa-e4d2658480ee-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj\" (UID: \"8535bf4a-f163-47e2-8baa-e4d2658480ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj"
Sep 08 13:46:02.790865 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:02.790836    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4xzq\" (UniqueName: \"kubernetes.io/projected/8535bf4a-f163-47e2-8baa-e4d2658480ee-kube-api-access-g4xzq\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj\" (UID: \"8535bf4a-f163-47e2-8baa-e4d2658480ee\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj"
Sep 08 13:46:02.863237 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:02.863148    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj"
Sep 08 13:46:03.145206 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:03.145167    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj"]
Sep 08 13:46:03.147239 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:46:03.147208    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8535bf4a_f163_47e2_8baa_e4d2658480ee.slice/crio-b60c76388bffcedb3058149e65770a884d7e27834b59d71020e42a66f2574aed WatchSource:0}: Error finding container b60c76388bffcedb3058149e65770a884d7e27834b59d71020e42a66f2574aed: Status 404 returned error can't find the container with id b60c76388bffcedb3058149e65770a884d7e27834b59d71020e42a66f2574aed
Sep 08 13:46:04.068522 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:04.068470    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj" event={"ID":"8535bf4a-f163-47e2-8baa-e4d2658480ee","Type":"ContainerStarted","Data":"4ab71d23602a82e47bbfd63236019f32bc6899bb0b21e7da77a1c5d37c0b42a7"}
Sep 08 13:46:04.068946 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:04.068566    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj" event={"ID":"8535bf4a-f163-47e2-8baa-e4d2658480ee","Type":"ContainerStarted","Data":"b60c76388bffcedb3058149e65770a884d7e27834b59d71020e42a66f2574aed"}
Sep 08 13:46:04.068946 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:04.068603    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj"
Sep 08 13:46:04.092436 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:04.092387    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj" podStartSLOduration=2.092373192 podStartE2EDuration="2.092373192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:46:04.09220407 +0000 UTC m=+1795.552848881" watchObservedRunningTime="2026-09-08 13:46:04.092373192 +0000 UTC m=+1795.553018001"
Sep 08 13:46:09.095778 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:09.095747    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 13:46:09.099072 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:09.099050    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 13:46:20.504961 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:20.504919    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"]
Sep 08 13:46:20.509688 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:20.509652    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"]
Sep 08 13:46:35.077474 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.077438    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj"
Sep 08 13:46:35.527210 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.527146    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4" podUID="cecd6812-086a-4788-a9d6-e0ad3bc83ddc" containerName="main" containerID="cri-o://2564777e05a881938f6d2e6c5197fac68c2581931810fa40834c61847c2b8a6c" gracePeriod=45
Sep 08 13:46:35.530426 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.530395    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh" podUID="3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714" containerName="main" containerID="cri-o://12489094cc9941f489b5c1cebd5ab3bdbc0f77d6651ea97591ff665d046f3546" gracePeriod=45
Sep 08 13:46:35.594546 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.594500    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4" podUID="cecd6812-086a-4788-a9d6-e0ad3bc83ddc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 08 13:46:35.637528 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.637483    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh" podUID="3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 08 13:46:35.807163 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.807135    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:46:35.810114 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.810083    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:46:35.884000 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.883955    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-tmp-dir\") pod \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") "
Sep 08 13:46:35.884206 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884027    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-model-cache\") pod \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") "
Sep 08 13:46:35.884206 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884081    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-dshm\") pod \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") "
Sep 08 13:46:35.884206 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884103    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-model-cache\") pod \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") "
Sep 08 13:46:35.884206 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884130    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-home\") pod \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") "
Sep 08 13:46:35.884206 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884159    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-tls-certs\") pod \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") "
Sep 08 13:46:35.884206 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884179    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-dshm\") pod \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") "
Sep 08 13:46:35.884558 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884221    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-home\") pod \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") "
Sep 08 13:46:35.884558 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884258    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bn88\" (UniqueName: \"kubernetes.io/projected/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-kube-api-access-9bn88\") pod \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") "
Sep 08 13:46:35.884558 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884337    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4928\" (UniqueName: \"kubernetes.io/projected/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-kube-api-access-c4928\") pod \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\" (UID: \"cecd6812-086a-4788-a9d6-e0ad3bc83ddc\") "
Sep 08 13:46:35.884558 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884377    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-tmp-dir\") pod \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") "
Sep 08 13:46:35.884558 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884351    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714" (UID: "3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:35.884558 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884403    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-tls-certs\") pod \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\" (UID: \"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714\") "
Sep 08 13:46:35.884558 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884403    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-model-cache" (OuterVolumeSpecName: "model-cache") pod "cecd6812-086a-4788-a9d6-e0ad3bc83ddc" (UID: "cecd6812-086a-4788-a9d6-e0ad3bc83ddc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:35.884558 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884427    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-home" (OuterVolumeSpecName: "home") pod "cecd6812-086a-4788-a9d6-e0ad3bc83ddc" (UID: "cecd6812-086a-4788-a9d6-e0ad3bc83ddc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:35.884957 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884596    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cecd6812-086a-4788-a9d6-e0ad3bc83ddc" (UID: "cecd6812-086a-4788-a9d6-e0ad3bc83ddc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:35.884957 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884618    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-home" (OuterVolumeSpecName: "home") pod "3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714" (UID: "3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:35.884957 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884733    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-tmp-dir\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:46:35.884957 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884793    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-model-cache\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:46:35.884957 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884807    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-model-cache\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:46:35.884957 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884819    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-home\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:46:35.884957 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884829    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-home\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:46:35.884957 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.884824    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714" (UID: "3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:35.887028 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.886994    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-dshm" (OuterVolumeSpecName: "dshm") pod "cecd6812-086a-4788-a9d6-e0ad3bc83ddc" (UID: "cecd6812-086a-4788-a9d6-e0ad3bc83ddc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:35.887156 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.887044    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cecd6812-086a-4788-a9d6-e0ad3bc83ddc" (UID: "cecd6812-086a-4788-a9d6-e0ad3bc83ddc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:46:35.887347 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.887320    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-dshm" (OuterVolumeSpecName: "dshm") pod "3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714" (UID: "3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:46:35.887347 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.887332    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714" (UID: "3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:46:35.887347 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.887343    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-kube-api-access-9bn88" (OuterVolumeSpecName: "kube-api-access-9bn88") pod "3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714" (UID: "3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714"). InnerVolumeSpecName "kube-api-access-9bn88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:46:35.887529 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.887330    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-kube-api-access-c4928" (OuterVolumeSpecName: "kube-api-access-c4928") pod "cecd6812-086a-4788-a9d6-e0ad3bc83ddc" (UID: "cecd6812-086a-4788-a9d6-e0ad3bc83ddc"). InnerVolumeSpecName "kube-api-access-c4928". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:46:35.985546 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.985505    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-dshm\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:46:35.985546 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.985536    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:46:35.985546 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.985547    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-dshm\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:46:35.985546 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.985555    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bn88\" (UniqueName: \"kubernetes.io/projected/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-kube-api-access-9bn88\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:46:35.985821 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.985566    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4928\" (UniqueName: \"kubernetes.io/projected/cecd6812-086a-4788-a9d6-e0ad3bc83ddc-kube-api-access-c4928\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:46:35.985821 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.985575    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-tmp-dir\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:46:35.985821 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:35.985583    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:46:36.251673 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:36.251638    2561 generic.go:356] "Generic (PLEG): container finished" podID="cecd6812-086a-4788-a9d6-e0ad3bc83ddc" containerID="2564777e05a881938f6d2e6c5197fac68c2581931810fa40834c61847c2b8a6c" exitCode=0
Sep 08 13:46:36.252121 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:36.251684    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"
Sep 08 13:46:36.252121 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:36.251725    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4" event={"ID":"cecd6812-086a-4788-a9d6-e0ad3bc83ddc","Type":"ContainerDied","Data":"2564777e05a881938f6d2e6c5197fac68c2581931810fa40834c61847c2b8a6c"}
Sep 08 13:46:36.252121 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:36.251782    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4" event={"ID":"cecd6812-086a-4788-a9d6-e0ad3bc83ddc","Type":"ContainerDied","Data":"9cd532b3011fc404f27599875a0a010fe5bf8e66e63f05f7d154223d88e84ce4"}
Sep 08 13:46:36.252121 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:36.251805    2561 scope.go:117] "RemoveContainer" containerID="2564777e05a881938f6d2e6c5197fac68c2581931810fa40834c61847c2b8a6c"
Sep 08 13:46:36.253328 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:36.253304    2561 generic.go:356] "Generic (PLEG): container finished" podID="3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714" containerID="12489094cc9941f489b5c1cebd5ab3bdbc0f77d6651ea97591ff665d046f3546" exitCode=0
Sep 08 13:46:36.253458 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:36.253343    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"
Sep 08 13:46:36.253458 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:36.253393    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh" event={"ID":"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714","Type":"ContainerDied","Data":"12489094cc9941f489b5c1cebd5ab3bdbc0f77d6651ea97591ff665d046f3546"}
Sep 08 13:46:36.253458 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:36.253436    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh" event={"ID":"3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714","Type":"ContainerDied","Data":"2cc3cb21f724f7b796faa452445cb0c9caf60acd9d30b3f9c273dbe6ae3a91a3"}
Sep 08 13:46:36.261125 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:36.260986    2561 scope.go:117] "RemoveContainer" containerID="2564777e05a881938f6d2e6c5197fac68c2581931810fa40834c61847c2b8a6c"
Sep 08 13:46:36.261389 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:46:36.261364    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2564777e05a881938f6d2e6c5197fac68c2581931810fa40834c61847c2b8a6c\": container with ID starting with 2564777e05a881938f6d2e6c5197fac68c2581931810fa40834c61847c2b8a6c not found: ID does not exist" containerID="2564777e05a881938f6d2e6c5197fac68c2581931810fa40834c61847c2b8a6c"
Sep 08 13:46:36.261462 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:36.261401    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2564777e05a881938f6d2e6c5197fac68c2581931810fa40834c61847c2b8a6c"} err="failed to get container status \"2564777e05a881938f6d2e6c5197fac68c2581931810fa40834c61847c2b8a6c\": rpc error: code = NotFound desc = could not find container \"2564777e05a881938f6d2e6c5197fac68c2581931810fa40834c61847c2b8a6c\": container with ID starting with 2564777e05a881938f6d2e6c5197fac68c2581931810fa40834c61847c2b8a6c not found: ID does not exist"
Sep 08 13:46:36.261462 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:36.261420    2561 scope.go:117] "RemoveContainer" containerID="12489094cc9941f489b5c1cebd5ab3bdbc0f77d6651ea97591ff665d046f3546"
Sep 08 13:46:36.269793 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:36.269775    2561 scope.go:117] "RemoveContainer" containerID="12489094cc9941f489b5c1cebd5ab3bdbc0f77d6651ea97591ff665d046f3546"
Sep 08 13:46:36.270084 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:46:36.270063    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12489094cc9941f489b5c1cebd5ab3bdbc0f77d6651ea97591ff665d046f3546\": container with ID starting with 12489094cc9941f489b5c1cebd5ab3bdbc0f77d6651ea97591ff665d046f3546 not found: ID does not exist" containerID="12489094cc9941f489b5c1cebd5ab3bdbc0f77d6651ea97591ff665d046f3546"
Sep 08 13:46:36.270167 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:36.270096    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12489094cc9941f489b5c1cebd5ab3bdbc0f77d6651ea97591ff665d046f3546"} err="failed to get container status \"12489094cc9941f489b5c1cebd5ab3bdbc0f77d6651ea97591ff665d046f3546\": rpc error: code = NotFound desc = could not find container \"12489094cc9941f489b5c1cebd5ab3bdbc0f77d6651ea97591ff665d046f3546\": container with ID starting with 12489094cc9941f489b5c1cebd5ab3bdbc0f77d6651ea97591ff665d046f3546 not found: ID does not exist"
Sep 08 13:46:36.279416 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:36.279384    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"]
Sep 08 13:46:36.283694 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:36.283660    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7rfjh"]
Sep 08 13:46:36.296904 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:36.296873    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"]
Sep 08 13:46:36.301111 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:36.301085    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-v99b4"]
Sep 08 13:46:37.007672 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:37.007640    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714" path="/var/lib/kubelet/pods/3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714/volumes"
Sep 08 13:46:37.008047 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:46:37.008034    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cecd6812-086a-4788-a9d6-e0ad3bc83ddc" path="/var/lib/kubelet/pods/cecd6812-086a-4788-a9d6-e0ad3bc83ddc/volumes"
Sep 08 13:47:12.261200 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:12.261161    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj"]
Sep 08 13:47:15.076513 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:15.076473    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj" podUID="8535bf4a-f163-47e2-8baa-e4d2658480ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:47:25.076275 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:25.076232    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj" podUID="8535bf4a-f163-47e2-8baa-e4d2658480ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:47:27.281735 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:27.281670    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj" podUID="8535bf4a-f163-47e2-8baa-e4d2658480ee" containerName="main" containerID="cri-o://4ab71d23602a82e47bbfd63236019f32bc6899bb0b21e7da77a1c5d37c0b42a7" gracePeriod=45
Sep 08 13:47:35.076590 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:35.076487    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj" podUID="8535bf4a-f163-47e2-8baa-e4d2658480ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:47:35.076975 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:35.076643    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj"
Sep 08 13:47:45.076515 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:45.076478    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj" podUID="8535bf4a-f163-47e2-8baa-e4d2658480ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:47:55.076224 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:55.076176    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj" podUID="8535bf4a-f163-47e2-8baa-e4d2658480ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:47:57.541092 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:57.541017    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj"
Sep 08 13:47:57.703024 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:57.702990    2561 generic.go:356] "Generic (PLEG): container finished" podID="8535bf4a-f163-47e2-8baa-e4d2658480ee" containerID="4ab71d23602a82e47bbfd63236019f32bc6899bb0b21e7da77a1c5d37c0b42a7" exitCode=0
Sep 08 13:47:57.703191 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:57.703030    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj"
Sep 08 13:47:57.703191 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:57.703067    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj" event={"ID":"8535bf4a-f163-47e2-8baa-e4d2658480ee","Type":"ContainerDied","Data":"4ab71d23602a82e47bbfd63236019f32bc6899bb0b21e7da77a1c5d37c0b42a7"}
Sep 08 13:47:57.703191 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:57.703123    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj" event={"ID":"8535bf4a-f163-47e2-8baa-e4d2658480ee","Type":"ContainerDied","Data":"b60c76388bffcedb3058149e65770a884d7e27834b59d71020e42a66f2574aed"}
Sep 08 13:47:57.703191 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:57.703145    2561 scope.go:117] "RemoveContainer" containerID="4ab71d23602a82e47bbfd63236019f32bc6899bb0b21e7da77a1c5d37c0b42a7"
Sep 08 13:47:57.714069 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:57.714052    2561 scope.go:117] "RemoveContainer" containerID="4ab71d23602a82e47bbfd63236019f32bc6899bb0b21e7da77a1c5d37c0b42a7"
Sep 08 13:47:57.714366 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:47:57.714348    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ab71d23602a82e47bbfd63236019f32bc6899bb0b21e7da77a1c5d37c0b42a7\": container with ID starting with 4ab71d23602a82e47bbfd63236019f32bc6899bb0b21e7da77a1c5d37c0b42a7 not found: ID does not exist" containerID="4ab71d23602a82e47bbfd63236019f32bc6899bb0b21e7da77a1c5d37c0b42a7"
Sep 08 13:47:57.714419 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:57.714373    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ab71d23602a82e47bbfd63236019f32bc6899bb0b21e7da77a1c5d37c0b42a7"} err="failed to get container status \"4ab71d23602a82e47bbfd63236019f32bc6899bb0b21e7da77a1c5d37c0b42a7\": rpc error: code = NotFound desc = could not find container \"4ab71d23602a82e47bbfd63236019f32bc6899bb0b21e7da77a1c5d37c0b42a7\": container with ID starting with 4ab71d23602a82e47bbfd63236019f32bc6899bb0b21e7da77a1c5d37c0b42a7 not found: ID does not exist"
Sep 08 13:47:57.722673 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:57.722651    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8535bf4a-f163-47e2-8baa-e4d2658480ee-tls-certs\") pod \"8535bf4a-f163-47e2-8baa-e4d2658480ee\" (UID: \"8535bf4a-f163-47e2-8baa-e4d2658480ee\") "
Sep 08 13:47:57.722779 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:57.722756    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4xzq\" (UniqueName: \"kubernetes.io/projected/8535bf4a-f163-47e2-8baa-e4d2658480ee-kube-api-access-g4xzq\") pod \"8535bf4a-f163-47e2-8baa-e4d2658480ee\" (UID: \"8535bf4a-f163-47e2-8baa-e4d2658480ee\") "
Sep 08 13:47:57.724922 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:57.724898    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8535bf4a-f163-47e2-8baa-e4d2658480ee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8535bf4a-f163-47e2-8baa-e4d2658480ee" (UID: "8535bf4a-f163-47e2-8baa-e4d2658480ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:47:57.725095 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:57.724977    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8535bf4a-f163-47e2-8baa-e4d2658480ee-kube-api-access-g4xzq" (OuterVolumeSpecName: "kube-api-access-g4xzq") pod "8535bf4a-f163-47e2-8baa-e4d2658480ee" (UID: "8535bf4a-f163-47e2-8baa-e4d2658480ee"). InnerVolumeSpecName "kube-api-access-g4xzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:47:57.823858 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:57.823773    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8535bf4a-f163-47e2-8baa-e4d2658480ee-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:47:57.823858 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:57.823801    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4xzq\" (UniqueName: \"kubernetes.io/projected/8535bf4a-f163-47e2-8baa-e4d2658480ee-kube-api-access-g4xzq\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:47:58.026187 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:58.026158    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj"]
Sep 08 13:47:58.031861 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:58.031837    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkfxjj"]
Sep 08 13:47:59.007062 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:47:59.007030    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8535bf4a-f163-47e2-8baa-e4d2658480ee" path="/var/lib/kubelet/pods/8535bf4a-f163-47e2-8baa-e4d2658480ee/volumes"
Sep 08 13:48:18.833831 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:18.833798    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh"]
Sep 08 13:48:18.834459 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:18.834335    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8535bf4a-f163-47e2-8baa-e4d2658480ee" containerName="main"
Sep 08 13:48:18.834459 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:18.834355    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8535bf4a-f163-47e2-8baa-e4d2658480ee" containerName="main"
Sep 08 13:48:18.834459 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:18.834368    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714" containerName="main"
Sep 08 13:48:18.834459 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:18.834377    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714" containerName="main"
Sep 08 13:48:18.834459 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:18.834413    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cecd6812-086a-4788-a9d6-e0ad3bc83ddc" containerName="main"
Sep 08 13:48:18.834459 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:18.834421    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cecd6812-086a-4788-a9d6-e0ad3bc83ddc" containerName="main"
Sep 08 13:48:18.834847 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:18.834511    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d1aeb9c-8b21-4ff3-b0c0-b7581eb13714" containerName="main"
Sep 08 13:48:18.834847 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:18.834522    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cecd6812-086a-4788-a9d6-e0ad3bc83ddc" containerName="main"
Sep 08 13:48:18.834847 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:18.834534    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8535bf4a-f163-47e2-8baa-e4d2658480ee" containerName="main"
Sep 08 13:48:18.837701 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:18.837679    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh"
Sep 08 13:48:18.840510 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:18.840480    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 08 13:48:18.841327 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:18.841303    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-pbbdj\""
Sep 08 13:48:18.841455 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:18.841353    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 08 13:48:18.841455 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:18.841408    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-pmmfl\""
Sep 08 13:48:18.841579 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:18.841355    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 08 13:48:18.857849 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:18.857817    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh"]
Sep 08 13:48:18.913769 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:18.913728    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49kc2\" (UniqueName: \"kubernetes.io/projected/5deaa079-a3dd-4af5-b183-b55aeaf431ef-kube-api-access-49kc2\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh\" (UID: \"5deaa079-a3dd-4af5-b183-b55aeaf431ef\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh"
Sep 08 13:48:18.913963 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:18.913895    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5deaa079-a3dd-4af5-b183-b55aeaf431ef-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh\" (UID: \"5deaa079-a3dd-4af5-b183-b55aeaf431ef\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh"
Sep 08 13:48:19.014342 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:19.014308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5deaa079-a3dd-4af5-b183-b55aeaf431ef-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh\" (UID: \"5deaa079-a3dd-4af5-b183-b55aeaf431ef\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh"
Sep 08 13:48:19.014535 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:19.014357    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49kc2\" (UniqueName: \"kubernetes.io/projected/5deaa079-a3dd-4af5-b183-b55aeaf431ef-kube-api-access-49kc2\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh\" (UID: \"5deaa079-a3dd-4af5-b183-b55aeaf431ef\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh"
Sep 08 13:48:19.016815 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:19.016790    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5deaa079-a3dd-4af5-b183-b55aeaf431ef-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh\" (UID: \"5deaa079-a3dd-4af5-b183-b55aeaf431ef\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh"
Sep 08 13:48:19.024641 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:19.024614    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49kc2\" (UniqueName: \"kubernetes.io/projected/5deaa079-a3dd-4af5-b183-b55aeaf431ef-kube-api-access-49kc2\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh\" (UID: \"5deaa079-a3dd-4af5-b183-b55aeaf431ef\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh"
Sep 08 13:48:19.148871 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:19.148769    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh"
Sep 08 13:48:19.426952 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:19.426920    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh"]
Sep 08 13:48:19.429317 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:48:19.429270    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5deaa079_a3dd_4af5_b183_b55aeaf431ef.slice/crio-b2b548e686622d14a4736013b93a0e20e7d14e2525479eeb19c235f8976007fe WatchSource:0}: Error finding container b2b548e686622d14a4736013b93a0e20e7d14e2525479eeb19c235f8976007fe: Status 404 returned error can't find the container with id b2b548e686622d14a4736013b93a0e20e7d14e2525479eeb19c235f8976007fe
Sep 08 13:48:19.819832 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:19.819790    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh" event={"ID":"5deaa079-a3dd-4af5-b183-b55aeaf431ef","Type":"ContainerStarted","Data":"7d8acb545fa1e6ca0358993bf048fc3db680126368e87054c6e426fb1279ab09"}
Sep 08 13:48:19.820012 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:19.819882    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh" event={"ID":"5deaa079-a3dd-4af5-b183-b55aeaf431ef","Type":"ContainerStarted","Data":"b2b548e686622d14a4736013b93a0e20e7d14e2525479eeb19c235f8976007fe"}
Sep 08 13:48:19.820012 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:19.819908    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh"
Sep 08 13:48:19.839917 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:19.839853    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh" podStartSLOduration=1.839838437 podStartE2EDuration="1.839838437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:48:19.836774907 +0000 UTC m=+1931.297419717" watchObservedRunningTime="2026-09-08 13:48:19.839838437 +0000 UTC m=+1931.300483247"
Sep 08 13:48:44.898330 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:44.898277    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"]
Sep 08 13:48:44.901682 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:44.901657    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:44.904237 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:44.904214    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-xhxgn\""
Sep 08 13:48:44.905203 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:44.905081    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 08 13:48:44.905203 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:44.905103    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 08 13:48:44.905203 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:44.905118    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v2-kserve-self-signed-certs\""
Sep 08 13:48:44.916213 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:44.916188    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"]
Sep 08 13:48:44.952194 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:44.952163    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4738f049-1eef-4266-a26b-3d548cf48f5c-tls-certs\") pod \"stop-v2-kserve-7f68747c45-ktjdh\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:44.952194 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:44.952199    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-ktjdh\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:44.952521 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:44.952268    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-dshm\") pod \"stop-v2-kserve-7f68747c45-ktjdh\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:44.952521 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:44.952310    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-home\") pod \"stop-v2-kserve-7f68747c45-ktjdh\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:44.952521 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:44.952331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjcx2\" (UniqueName: \"kubernetes.io/projected/4738f049-1eef-4266-a26b-3d548cf48f5c-kube-api-access-fjcx2\") pod \"stop-v2-kserve-7f68747c45-ktjdh\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:44.952521 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:44.952373    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-model-cache\") pod \"stop-v2-kserve-7f68747c45-ktjdh\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:45.052955 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:45.052918    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-model-cache\") pod \"stop-v2-kserve-7f68747c45-ktjdh\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:45.053149 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:45.052975    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4738f049-1eef-4266-a26b-3d548cf48f5c-tls-certs\") pod \"stop-v2-kserve-7f68747c45-ktjdh\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:45.053149 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:45.052995    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-ktjdh\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:45.053149 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:45.053030    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-dshm\") pod \"stop-v2-kserve-7f68747c45-ktjdh\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:45.053149 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:45.053052    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-home\") pod \"stop-v2-kserve-7f68747c45-ktjdh\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:45.053149 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:45.053071    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjcx2\" (UniqueName: \"kubernetes.io/projected/4738f049-1eef-4266-a26b-3d548cf48f5c-kube-api-access-fjcx2\") pod \"stop-v2-kserve-7f68747c45-ktjdh\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:45.053451 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:45.053423    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-ktjdh\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:45.053515 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:45.053421    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-model-cache\") pod \"stop-v2-kserve-7f68747c45-ktjdh\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:45.053515 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:45.053505    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-home\") pod \"stop-v2-kserve-7f68747c45-ktjdh\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:45.055520 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:45.055489    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-dshm\") pod \"stop-v2-kserve-7f68747c45-ktjdh\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:45.055656 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:45.055634    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4738f049-1eef-4266-a26b-3d548cf48f5c-tls-certs\") pod \"stop-v2-kserve-7f68747c45-ktjdh\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:45.062979 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:45.062950    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjcx2\" (UniqueName: \"kubernetes.io/projected/4738f049-1eef-4266-a26b-3d548cf48f5c-kube-api-access-fjcx2\") pod \"stop-v2-kserve-7f68747c45-ktjdh\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:45.214936 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:45.214895    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:45.496437 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:45.496411    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"]
Sep 08 13:48:45.498341 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:48:45.498309    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4738f049_1eef_4266_a26b_3d548cf48f5c.slice/crio-4cd25e30378ec29562f7020556fe8710f2acaed2d27d462a55c098636553701d WatchSource:0}: Error finding container 4cd25e30378ec29562f7020556fe8710f2acaed2d27d462a55c098636553701d: Status 404 returned error can't find the container with id 4cd25e30378ec29562f7020556fe8710f2acaed2d27d462a55c098636553701d
Sep 08 13:48:45.966037 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:45.965997    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh" event={"ID":"4738f049-1eef-4266-a26b-3d548cf48f5c","Type":"ContainerStarted","Data":"f7be8f69f60eed193457e3fee3d43a1cab330dede171de3221ca1eaa666dbb18"}
Sep 08 13:48:45.966484 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:45.966062    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh" event={"ID":"4738f049-1eef-4266-a26b-3d548cf48f5c","Type":"ContainerStarted","Data":"4cd25e30378ec29562f7020556fe8710f2acaed2d27d462a55c098636553701d"}
Sep 08 13:48:45.990038 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:45.989982    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh" podStartSLOduration=1.989963762 podStartE2EDuration="1.989963762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:48:45.987204301 +0000 UTC m=+1957.447849112" watchObservedRunningTime="2026-09-08 13:48:45.989963762 +0000 UTC m=+1957.450608571"
Sep 08 13:48:46.215977 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:46.215934    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:50.827587 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:50.827555    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh"
Sep 08 13:48:55.215134 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:55.215089    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:55.231448 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:55.231410    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:48:56.031588 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:48:56.031556    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:49:31.119488 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:31.119453    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"]
Sep 08 13:49:42.648520 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:42.648478    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh"]
Sep 08 13:49:46.138900 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.138853    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh" podUID="4738f049-1eef-4266-a26b-3d548cf48f5c" containerName="main" containerID="cri-o://f7be8f69f60eed193457e3fee3d43a1cab330dede171de3221ca1eaa666dbb18" gracePeriod=45
Sep 08 13:49:46.290899 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.290868    2561 generic.go:356] "Generic (PLEG): container finished" podID="4738f049-1eef-4266-a26b-3d548cf48f5c" containerID="f7be8f69f60eed193457e3fee3d43a1cab330dede171de3221ca1eaa666dbb18" exitCode=0
Sep 08 13:49:46.291081 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.290939    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh" event={"ID":"4738f049-1eef-4266-a26b-3d548cf48f5c","Type":"ContainerDied","Data":"f7be8f69f60eed193457e3fee3d43a1cab330dede171de3221ca1eaa666dbb18"}
Sep 08 13:49:46.400267 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.400198    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:49:46.507114 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.507075    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-model-cache\") pod \"4738f049-1eef-4266-a26b-3d548cf48f5c\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") "
Sep 08 13:49:46.507406 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.507133    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-home\") pod \"4738f049-1eef-4266-a26b-3d548cf48f5c\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") "
Sep 08 13:49:46.507406 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.507167    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjcx2\" (UniqueName: \"kubernetes.io/projected/4738f049-1eef-4266-a26b-3d548cf48f5c-kube-api-access-fjcx2\") pod \"4738f049-1eef-4266-a26b-3d548cf48f5c\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") "
Sep 08 13:49:46.507406 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.507210    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-tmp-dir\") pod \"4738f049-1eef-4266-a26b-3d548cf48f5c\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") "
Sep 08 13:49:46.507406 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.507262    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4738f049-1eef-4266-a26b-3d548cf48f5c-tls-certs\") pod \"4738f049-1eef-4266-a26b-3d548cf48f5c\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") "
Sep 08 13:49:46.507406 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.507315    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-dshm\") pod \"4738f049-1eef-4266-a26b-3d548cf48f5c\" (UID: \"4738f049-1eef-4266-a26b-3d548cf48f5c\") "
Sep 08 13:49:46.507645 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.507426    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-model-cache" (OuterVolumeSpecName: "model-cache") pod "4738f049-1eef-4266-a26b-3d548cf48f5c" (UID: "4738f049-1eef-4266-a26b-3d548cf48f5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:46.507645 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.507484    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4738f049-1eef-4266-a26b-3d548cf48f5c" (UID: "4738f049-1eef-4266-a26b-3d548cf48f5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:46.507645 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.507503    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-home" (OuterVolumeSpecName: "home") pod "4738f049-1eef-4266-a26b-3d548cf48f5c" (UID: "4738f049-1eef-4266-a26b-3d548cf48f5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:46.507804 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.507653    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-model-cache\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:49:46.507804 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.507664    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-home\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:49:46.507804 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.507672    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-tmp-dir\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:49:46.509579 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.509550    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-dshm" (OuterVolumeSpecName: "dshm") pod "4738f049-1eef-4266-a26b-3d548cf48f5c" (UID: "4738f049-1eef-4266-a26b-3d548cf48f5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:46.509721 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.509689    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4738f049-1eef-4266-a26b-3d548cf48f5c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4738f049-1eef-4266-a26b-3d548cf48f5c" (UID: "4738f049-1eef-4266-a26b-3d548cf48f5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:49:46.509765 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.509753    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4738f049-1eef-4266-a26b-3d548cf48f5c-kube-api-access-fjcx2" (OuterVolumeSpecName: "kube-api-access-fjcx2") pod "4738f049-1eef-4266-a26b-3d548cf48f5c" (UID: "4738f049-1eef-4266-a26b-3d548cf48f5c"). InnerVolumeSpecName "kube-api-access-fjcx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:49:46.609157 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.609117    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4738f049-1eef-4266-a26b-3d548cf48f5c-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:49:46.609157 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.609152    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4738f049-1eef-4266-a26b-3d548cf48f5c-dshm\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:49:46.609157 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:46.609161    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjcx2\" (UniqueName: \"kubernetes.io/projected/4738f049-1eef-4266-a26b-3d548cf48f5c-kube-api-access-fjcx2\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:49:47.297828 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:47.297735    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh" event={"ID":"4738f049-1eef-4266-a26b-3d548cf48f5c","Type":"ContainerDied","Data":"4cd25e30378ec29562f7020556fe8710f2acaed2d27d462a55c098636553701d"}
Sep 08 13:49:47.297828 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:47.297771    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"
Sep 08 13:49:47.297828 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:47.297828    2561 scope.go:117] "RemoveContainer" containerID="f7be8f69f60eed193457e3fee3d43a1cab330dede171de3221ca1eaa666dbb18"
Sep 08 13:49:47.319754 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:47.319720    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"]
Sep 08 13:49:47.326728 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:47.326702    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-ktjdh"]
Sep 08 13:49:49.007017 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:49.006985    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4738f049-1eef-4266-a26b-3d548cf48f5c" path="/var/lib/kubelet/pods/4738f049-1eef-4266-a26b-3d548cf48f5c/volumes"
Sep 08 13:49:50.826961 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:50.826922    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh" podUID="5deaa079-a3dd-4af5-b183-b55aeaf431ef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:49:57.676665 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:49:57.676595    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh" podUID="5deaa079-a3dd-4af5-b183-b55aeaf431ef" containerName="main" containerID="cri-o://7d8acb545fa1e6ca0358993bf048fc3db680126368e87054c6e426fb1279ab09" gracePeriod=45
Sep 08 13:50:00.826778 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:00.826740    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh" podUID="5deaa079-a3dd-4af5-b183-b55aeaf431ef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:50:10.827449 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:10.827407    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh" podUID="5deaa079-a3dd-4af5-b183-b55aeaf431ef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:50:10.827866 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:10.827511    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh"
Sep 08 13:50:20.827466 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:20.827381    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh" podUID="5deaa079-a3dd-4af5-b183-b55aeaf431ef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:50:27.935754 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:27.935732    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh"
Sep 08 13:50:27.982552 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:27.982520    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5deaa079-a3dd-4af5-b183-b55aeaf431ef-tls-certs\") pod \"5deaa079-a3dd-4af5-b183-b55aeaf431ef\" (UID: \"5deaa079-a3dd-4af5-b183-b55aeaf431ef\") "
Sep 08 13:50:27.982745 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:27.982575    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49kc2\" (UniqueName: \"kubernetes.io/projected/5deaa079-a3dd-4af5-b183-b55aeaf431ef-kube-api-access-49kc2\") pod \"5deaa079-a3dd-4af5-b183-b55aeaf431ef\" (UID: \"5deaa079-a3dd-4af5-b183-b55aeaf431ef\") "
Sep 08 13:50:27.984855 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:27.984825    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5deaa079-a3dd-4af5-b183-b55aeaf431ef-kube-api-access-49kc2" (OuterVolumeSpecName: "kube-api-access-49kc2") pod "5deaa079-a3dd-4af5-b183-b55aeaf431ef" (UID: "5deaa079-a3dd-4af5-b183-b55aeaf431ef"). InnerVolumeSpecName "kube-api-access-49kc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:50:27.984964 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:27.984831    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5deaa079-a3dd-4af5-b183-b55aeaf431ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5deaa079-a3dd-4af5-b183-b55aeaf431ef" (UID: "5deaa079-a3dd-4af5-b183-b55aeaf431ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:50:28.083256 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:28.083217    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5deaa079-a3dd-4af5-b183-b55aeaf431ef-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:50:28.083256 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:28.083251    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49kc2\" (UniqueName: \"kubernetes.io/projected/5deaa079-a3dd-4af5-b183-b55aeaf431ef-kube-api-access-49kc2\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:50:28.523338 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:28.523271    2561 generic.go:356] "Generic (PLEG): container finished" podID="5deaa079-a3dd-4af5-b183-b55aeaf431ef" containerID="7d8acb545fa1e6ca0358993bf048fc3db680126368e87054c6e426fb1279ab09" exitCode=0
Sep 08 13:50:28.523512 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:28.523354    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh"
Sep 08 13:50:28.523512 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:28.523358    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh" event={"ID":"5deaa079-a3dd-4af5-b183-b55aeaf431ef","Type":"ContainerDied","Data":"7d8acb545fa1e6ca0358993bf048fc3db680126368e87054c6e426fb1279ab09"}
Sep 08 13:50:28.523512 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:28.523415    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh" event={"ID":"5deaa079-a3dd-4af5-b183-b55aeaf431ef","Type":"ContainerDied","Data":"b2b548e686622d14a4736013b93a0e20e7d14e2525479eeb19c235f8976007fe"}
Sep 08 13:50:28.523512 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:28.523439    2561 scope.go:117] "RemoveContainer" containerID="7d8acb545fa1e6ca0358993bf048fc3db680126368e87054c6e426fb1279ab09"
Sep 08 13:50:28.532703 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:28.532685    2561 scope.go:117] "RemoveContainer" containerID="7d8acb545fa1e6ca0358993bf048fc3db680126368e87054c6e426fb1279ab09"
Sep 08 13:50:28.532981 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:50:28.532961    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d8acb545fa1e6ca0358993bf048fc3db680126368e87054c6e426fb1279ab09\": container with ID starting with 7d8acb545fa1e6ca0358993bf048fc3db680126368e87054c6e426fb1279ab09 not found: ID does not exist" containerID="7d8acb545fa1e6ca0358993bf048fc3db680126368e87054c6e426fb1279ab09"
Sep 08 13:50:28.533045 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:28.532995    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d8acb545fa1e6ca0358993bf048fc3db680126368e87054c6e426fb1279ab09"} err="failed to get container status \"7d8acb545fa1e6ca0358993bf048fc3db680126368e87054c6e426fb1279ab09\": rpc error: code = NotFound desc = could not find container \"7d8acb545fa1e6ca0358993bf048fc3db680126368e87054c6e426fb1279ab09\": container with ID starting with 7d8acb545fa1e6ca0358993bf048fc3db680126368e87054c6e426fb1279ab09 not found: ID does not exist"
Sep 08 13:50:28.553824 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:28.553791    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh"]
Sep 08 13:50:28.557673 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:28.557642    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sh5lh"]
Sep 08 13:50:29.012614 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:29.012580    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5deaa079-a3dd-4af5-b183-b55aeaf431ef" path="/var/lib/kubelet/pods/5deaa079-a3dd-4af5-b183-b55aeaf431ef/volumes"
Sep 08 13:50:50.837737 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:50.837696    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7"]
Sep 08 13:50:50.838330 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:50.838307    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5deaa079-a3dd-4af5-b183-b55aeaf431ef" containerName="main"
Sep 08 13:50:50.838384 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:50.838334    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5deaa079-a3dd-4af5-b183-b55aeaf431ef" containerName="main"
Sep 08 13:50:50.838384 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:50.838349    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4738f049-1eef-4266-a26b-3d548cf48f5c" containerName="main"
Sep 08 13:50:50.838384 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:50.838358    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4738f049-1eef-4266-a26b-3d548cf48f5c" containerName="main"
Sep 08 13:50:50.838483 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:50.838473    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5deaa079-a3dd-4af5-b183-b55aeaf431ef" containerName="main"
Sep 08 13:50:50.838521 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:50.838486    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4738f049-1eef-4266-a26b-3d548cf48f5c" containerName="main"
Sep 08 13:50:50.841846 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:50.841820    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7"
Sep 08 13:50:50.845341 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:50.845272    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-6r98f\""
Sep 08 13:50:50.845341 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:50.845325    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 08 13:50:50.845610 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:50.845422    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-49r6v\""
Sep 08 13:50:50.845610 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:50.845272    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 08 13:50:50.845610 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:50.845465    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 08 13:50:50.853944 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:50.853913    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7"]
Sep 08 13:50:50.881680 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:50.881640    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22rv6\" (UniqueName: \"kubernetes.io/projected/e16e6f4a-11ba-4d15-84cc-9f24923aa21d-kube-api-access-22rv6\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7\" (UID: \"e16e6f4a-11ba-4d15-84cc-9f24923aa21d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7"
Sep 08 13:50:50.881850 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:50.881774    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e16e6f4a-11ba-4d15-84cc-9f24923aa21d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7\" (UID: \"e16e6f4a-11ba-4d15-84cc-9f24923aa21d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7"
Sep 08 13:50:50.982950 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:50.982913    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e16e6f4a-11ba-4d15-84cc-9f24923aa21d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7\" (UID: \"e16e6f4a-11ba-4d15-84cc-9f24923aa21d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7"
Sep 08 13:50:50.983110 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:50.982979    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22rv6\" (UniqueName: \"kubernetes.io/projected/e16e6f4a-11ba-4d15-84cc-9f24923aa21d-kube-api-access-22rv6\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7\" (UID: \"e16e6f4a-11ba-4d15-84cc-9f24923aa21d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7"
Sep 08 13:50:50.985635 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:50.985608    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e16e6f4a-11ba-4d15-84cc-9f24923aa21d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7\" (UID: \"e16e6f4a-11ba-4d15-84cc-9f24923aa21d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7"
Sep 08 13:50:50.991508 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:50.991475    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22rv6\" (UniqueName: \"kubernetes.io/projected/e16e6f4a-11ba-4d15-84cc-9f24923aa21d-kube-api-access-22rv6\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7\" (UID: \"e16e6f4a-11ba-4d15-84cc-9f24923aa21d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7"
Sep 08 13:50:51.154886 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:51.154794    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7"
Sep 08 13:50:51.431594 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:51.431569    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7"]
Sep 08 13:50:51.433778 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:50:51.433750    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16e6f4a_11ba_4d15_84cc_9f24923aa21d.slice/crio-d5242a6396d22a3a0228134a74982f8cf0c7081e071632ae93699f4d0cf7f0c7 WatchSource:0}: Error finding container d5242a6396d22a3a0228134a74982f8cf0c7081e071632ae93699f4d0cf7f0c7: Status 404 returned error can't find the container with id d5242a6396d22a3a0228134a74982f8cf0c7081e071632ae93699f4d0cf7f0c7
Sep 08 13:50:51.647675 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:51.647640    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7" event={"ID":"e16e6f4a-11ba-4d15-84cc-9f24923aa21d","Type":"ContainerStarted","Data":"5a0e0ef3e1372450d7b3db0950bcf2e991ac9a5fe57b5bcc21eba2623b8a0a91"}
Sep 08 13:50:51.647898 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:51.647702    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7" event={"ID":"e16e6f4a-11ba-4d15-84cc-9f24923aa21d","Type":"ContainerStarted","Data":"d5242a6396d22a3a0228134a74982f8cf0c7081e071632ae93699f4d0cf7f0c7"}
Sep 08 13:50:51.647898 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:51.647784    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7"
Sep 08 13:50:51.668704 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:50:51.668649    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7" podStartSLOduration=1.668631349 podStartE2EDuration="1.668631349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:50:51.667369754 +0000 UTC m=+2083.128014564" watchObservedRunningTime="2026-09-08 13:50:51.668631349 +0000 UTC m=+2083.129276161"
Sep 08 13:51:09.130773 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:09.130745    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 13:51:09.135736 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:09.135715    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 13:51:22.654697 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:22.654665    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7"
Sep 08 13:51:40.315205 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.315163    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"]
Sep 08 13:51:40.319232 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.319204    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:40.322515 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.322490    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-ht47h\""
Sep 08 13:51:40.322632 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.322490    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\""
Sep 08 13:51:40.322694 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.322683    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 08 13:51:40.322785 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.322767    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 08 13:51:40.330503 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.330479    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"]
Sep 08 13:51:40.427129 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.427088    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5h9b\" (UniqueName: \"kubernetes.io/projected/f995836b-adf2-4ded-affb-0d91225898b5-kube-api-access-k5h9b\") pod \"leave-v1-kserve-65684d4f5-5z84f\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:40.427129 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.427131    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-model-cache\") pod \"leave-v1-kserve-65684d4f5-5z84f\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:40.427394 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.427197    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-5z84f\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:40.427394 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.427241    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-dshm\") pod \"leave-v1-kserve-65684d4f5-5z84f\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:40.427394 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.427258    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f995836b-adf2-4ded-affb-0d91225898b5-tls-certs\") pod \"leave-v1-kserve-65684d4f5-5z84f\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:40.427551 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.427403    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-home\") pod \"leave-v1-kserve-65684d4f5-5z84f\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:40.528756 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.528716    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-home\") pod \"leave-v1-kserve-65684d4f5-5z84f\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:40.528960 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.528781    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5h9b\" (UniqueName: \"kubernetes.io/projected/f995836b-adf2-4ded-affb-0d91225898b5-kube-api-access-k5h9b\") pod \"leave-v1-kserve-65684d4f5-5z84f\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:40.528960 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.528801    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-model-cache\") pod \"leave-v1-kserve-65684d4f5-5z84f\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:40.528960 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.528821    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-5z84f\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:40.528960 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.528849    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-dshm\") pod \"leave-v1-kserve-65684d4f5-5z84f\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:40.528960 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.528864    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f995836b-adf2-4ded-affb-0d91225898b5-tls-certs\") pod \"leave-v1-kserve-65684d4f5-5z84f\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:40.529313 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.529265    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-home\") pod \"leave-v1-kserve-65684d4f5-5z84f\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:40.529391 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.529297    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-model-cache\") pod \"leave-v1-kserve-65684d4f5-5z84f\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:40.529391 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.529367    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-5z84f\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:40.531383 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.531360    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-dshm\") pod \"leave-v1-kserve-65684d4f5-5z84f\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:40.531540 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.531517    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f995836b-adf2-4ded-affb-0d91225898b5-tls-certs\") pod \"leave-v1-kserve-65684d4f5-5z84f\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:40.539052 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.539030    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5h9b\" (UniqueName: \"kubernetes.io/projected/f995836b-adf2-4ded-affb-0d91225898b5-kube-api-access-k5h9b\") pod \"leave-v1-kserve-65684d4f5-5z84f\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:40.630655 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.630567    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:40.920322 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:40.920297    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"]
Sep 08 13:51:40.922847 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:51:40.922823    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf995836b_adf2_4ded_affb_0d91225898b5.slice/crio-98a28a5411a86f05515a034b250aecb407c7d4165396da1e8c886868e8f63ec1 WatchSource:0}: Error finding container 98a28a5411a86f05515a034b250aecb407c7d4165396da1e8c886868e8f63ec1: Status 404 returned error can't find the container with id 98a28a5411a86f05515a034b250aecb407c7d4165396da1e8c886868e8f63ec1
Sep 08 13:51:41.929641 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:41.929598    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f" event={"ID":"f995836b-adf2-4ded-affb-0d91225898b5","Type":"ContainerStarted","Data":"ac1e5f3736c167e45843079652a5d471666f6871d5c432f061de51d959698215"}
Sep 08 13:51:41.930113 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:41.929660    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f" event={"ID":"f995836b-adf2-4ded-affb-0d91225898b5","Type":"ContainerStarted","Data":"98a28a5411a86f05515a034b250aecb407c7d4165396da1e8c886868e8f63ec1"}
Sep 08 13:51:41.952961 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:41.952908    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f" podStartSLOduration=1.952894073 podStartE2EDuration="1.952894073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:51:41.950866707 +0000 UTC m=+2133.411511545" watchObservedRunningTime="2026-09-08 13:51:41.952894073 +0000 UTC m=+2133.413538882"
Sep 08 13:51:42.631644 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:42.631601    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:50.631112 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:50.631074    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:50.644317 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:50.644263    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:51:50.994758 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:51:50.994728    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:52:04.749157 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:04.749059    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7"]
Sep 08 13:52:10.719000 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:10.718950    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"]
Sep 08 13:52:12.655412 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:12.655370    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7" podUID="e16e6f4a-11ba-4d15-84cc-9f24923aa21d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:52:19.772124 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:19.772066    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7" podUID="e16e6f4a-11ba-4d15-84cc-9f24923aa21d" containerName="main" containerID="cri-o://5a0e0ef3e1372450d7b3db0950bcf2e991ac9a5fe57b5bcc21eba2623b8a0a91" gracePeriod=45
Sep 08 13:52:22.655394 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:22.655352    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7" podUID="e16e6f4a-11ba-4d15-84cc-9f24923aa21d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:52:25.744046 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:25.744001    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f" podUID="f995836b-adf2-4ded-affb-0d91225898b5" containerName="main" containerID="cri-o://ac1e5f3736c167e45843079652a5d471666f6871d5c432f061de51d959698215" gracePeriod=45
Sep 08 13:52:26.001979 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.001907    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:52:26.152923 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.152891    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-model-cache\") pod \"f995836b-adf2-4ded-affb-0d91225898b5\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") "
Sep 08 13:52:26.153084 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.152934    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-dshm\") pod \"f995836b-adf2-4ded-affb-0d91225898b5\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") "
Sep 08 13:52:26.153084 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.153018    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f995836b-adf2-4ded-affb-0d91225898b5-tls-certs\") pod \"f995836b-adf2-4ded-affb-0d91225898b5\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") "
Sep 08 13:52:26.153084 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.153059    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5h9b\" (UniqueName: \"kubernetes.io/projected/f995836b-adf2-4ded-affb-0d91225898b5-kube-api-access-k5h9b\") pod \"f995836b-adf2-4ded-affb-0d91225898b5\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") "
Sep 08 13:52:26.153250 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.153101    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-tmp-dir\") pod \"f995836b-adf2-4ded-affb-0d91225898b5\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") "
Sep 08 13:52:26.153250 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.153199    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-model-cache" (OuterVolumeSpecName: "model-cache") pod "f995836b-adf2-4ded-affb-0d91225898b5" (UID: "f995836b-adf2-4ded-affb-0d91225898b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:26.153400 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.153304    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-home\") pod \"f995836b-adf2-4ded-affb-0d91225898b5\" (UID: \"f995836b-adf2-4ded-affb-0d91225898b5\") "
Sep 08 13:52:26.153400 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.153334    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f995836b-adf2-4ded-affb-0d91225898b5" (UID: "f995836b-adf2-4ded-affb-0d91225898b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:26.153538 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.153511    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-home" (OuterVolumeSpecName: "home") pod "f995836b-adf2-4ded-affb-0d91225898b5" (UID: "f995836b-adf2-4ded-affb-0d91225898b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:26.153741 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.153671    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-tmp-dir\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:52:26.153741 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.153694    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-home\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:52:26.153741 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.153708    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-model-cache\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:52:26.155253 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.155224    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-dshm" (OuterVolumeSpecName: "dshm") pod "f995836b-adf2-4ded-affb-0d91225898b5" (UID: "f995836b-adf2-4ded-affb-0d91225898b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:26.155401 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.155317    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f995836b-adf2-4ded-affb-0d91225898b5-kube-api-access-k5h9b" (OuterVolumeSpecName: "kube-api-access-k5h9b") pod "f995836b-adf2-4ded-affb-0d91225898b5" (UID: "f995836b-adf2-4ded-affb-0d91225898b5"). InnerVolumeSpecName "kube-api-access-k5h9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:52:26.155401 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.155325    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f995836b-adf2-4ded-affb-0d91225898b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f995836b-adf2-4ded-affb-0d91225898b5" (UID: "f995836b-adf2-4ded-affb-0d91225898b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:52:26.177732 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.177708    2561 generic.go:356] "Generic (PLEG): container finished" podID="f995836b-adf2-4ded-affb-0d91225898b5" containerID="ac1e5f3736c167e45843079652a5d471666f6871d5c432f061de51d959698215" exitCode=0
Sep 08 13:52:26.177849 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.177762    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"
Sep 08 13:52:26.177849 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.177791    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f" event={"ID":"f995836b-adf2-4ded-affb-0d91225898b5","Type":"ContainerDied","Data":"ac1e5f3736c167e45843079652a5d471666f6871d5c432f061de51d959698215"}
Sep 08 13:52:26.177849 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.177846    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f" event={"ID":"f995836b-adf2-4ded-affb-0d91225898b5","Type":"ContainerDied","Data":"98a28a5411a86f05515a034b250aecb407c7d4165396da1e8c886868e8f63ec1"}
Sep 08 13:52:26.177952 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.177868    2561 scope.go:117] "RemoveContainer" containerID="ac1e5f3736c167e45843079652a5d471666f6871d5c432f061de51d959698215"
Sep 08 13:52:26.187048 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.187029    2561 scope.go:117] "RemoveContainer" containerID="ac1e5f3736c167e45843079652a5d471666f6871d5c432f061de51d959698215"
Sep 08 13:52:26.187373 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:52:26.187353    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1e5f3736c167e45843079652a5d471666f6871d5c432f061de51d959698215\": container with ID starting with ac1e5f3736c167e45843079652a5d471666f6871d5c432f061de51d959698215 not found: ID does not exist" containerID="ac1e5f3736c167e45843079652a5d471666f6871d5c432f061de51d959698215"
Sep 08 13:52:26.187449 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.187386    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1e5f3736c167e45843079652a5d471666f6871d5c432f061de51d959698215"} err="failed to get container status \"ac1e5f3736c167e45843079652a5d471666f6871d5c432f061de51d959698215\": rpc error: code = NotFound desc = could not find container \"ac1e5f3736c167e45843079652a5d471666f6871d5c432f061de51d959698215\": container with ID starting with ac1e5f3736c167e45843079652a5d471666f6871d5c432f061de51d959698215 not found: ID does not exist"
Sep 08 13:52:26.201701 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.201672    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"]
Sep 08 13:52:26.203806 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.203784    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f"]
Sep 08 13:52:26.255201 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.255112    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f995836b-adf2-4ded-affb-0d91225898b5-dshm\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:52:26.255201 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.255144    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f995836b-adf2-4ded-affb-0d91225898b5-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:52:26.255201 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.255154    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5h9b\" (UniqueName: \"kubernetes.io/projected/f995836b-adf2-4ded-affb-0d91225898b5-kube-api-access-k5h9b\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:52:26.984195 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:26.984147    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-5z84f" podUID="f995836b-adf2-4ded-affb-0d91225898b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 08 13:52:27.006497 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:27.006469    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f995836b-adf2-4ded-affb-0d91225898b5" path="/var/lib/kubelet/pods/f995836b-adf2-4ded-affb-0d91225898b5/volumes"
Sep 08 13:52:32.654909 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:32.654870    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7" podUID="e16e6f4a-11ba-4d15-84cc-9f24923aa21d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:52:32.655403 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:32.654985    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7"
Sep 08 13:52:42.655278 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:42.655241    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7" podUID="e16e6f4a-11ba-4d15-84cc-9f24923aa21d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:52:50.027704 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:50.027679    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7"
Sep 08 13:52:50.068370 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:50.068282    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22rv6\" (UniqueName: \"kubernetes.io/projected/e16e6f4a-11ba-4d15-84cc-9f24923aa21d-kube-api-access-22rv6\") pod \"e16e6f4a-11ba-4d15-84cc-9f24923aa21d\" (UID: \"e16e6f4a-11ba-4d15-84cc-9f24923aa21d\") "
Sep 08 13:52:50.068370 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:50.068330    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e16e6f4a-11ba-4d15-84cc-9f24923aa21d-tls-certs\") pod \"e16e6f4a-11ba-4d15-84cc-9f24923aa21d\" (UID: \"e16e6f4a-11ba-4d15-84cc-9f24923aa21d\") "
Sep 08 13:52:50.070579 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:50.070541    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16e6f4a-11ba-4d15-84cc-9f24923aa21d-kube-api-access-22rv6" (OuterVolumeSpecName: "kube-api-access-22rv6") pod "e16e6f4a-11ba-4d15-84cc-9f24923aa21d" (UID: "e16e6f4a-11ba-4d15-84cc-9f24923aa21d"). InnerVolumeSpecName "kube-api-access-22rv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:52:50.070707 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:50.070595    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e16e6f4a-11ba-4d15-84cc-9f24923aa21d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e16e6f4a-11ba-4d15-84cc-9f24923aa21d" (UID: "e16e6f4a-11ba-4d15-84cc-9f24923aa21d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:52:50.169514 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:50.169472    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22rv6\" (UniqueName: \"kubernetes.io/projected/e16e6f4a-11ba-4d15-84cc-9f24923aa21d-kube-api-access-22rv6\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:52:50.169514 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:50.169505    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e16e6f4a-11ba-4d15-84cc-9f24923aa21d-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:52:50.314450 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:50.314419    2561 generic.go:356] "Generic (PLEG): container finished" podID="e16e6f4a-11ba-4d15-84cc-9f24923aa21d" containerID="5a0e0ef3e1372450d7b3db0950bcf2e991ac9a5fe57b5bcc21eba2623b8a0a91" exitCode=0
Sep 08 13:52:50.314621 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:50.314467    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7"
Sep 08 13:52:50.314621 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:50.314503    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7" event={"ID":"e16e6f4a-11ba-4d15-84cc-9f24923aa21d","Type":"ContainerDied","Data":"5a0e0ef3e1372450d7b3db0950bcf2e991ac9a5fe57b5bcc21eba2623b8a0a91"}
Sep 08 13:52:50.314621 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:50.314559    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7" event={"ID":"e16e6f4a-11ba-4d15-84cc-9f24923aa21d","Type":"ContainerDied","Data":"d5242a6396d22a3a0228134a74982f8cf0c7081e071632ae93699f4d0cf7f0c7"}
Sep 08 13:52:50.314621 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:50.314583    2561 scope.go:117] "RemoveContainer" containerID="5a0e0ef3e1372450d7b3db0950bcf2e991ac9a5fe57b5bcc21eba2623b8a0a91"
Sep 08 13:52:50.323669 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:50.323651    2561 scope.go:117] "RemoveContainer" containerID="5a0e0ef3e1372450d7b3db0950bcf2e991ac9a5fe57b5bcc21eba2623b8a0a91"
Sep 08 13:52:50.323943 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:52:50.323922    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a0e0ef3e1372450d7b3db0950bcf2e991ac9a5fe57b5bcc21eba2623b8a0a91\": container with ID starting with 5a0e0ef3e1372450d7b3db0950bcf2e991ac9a5fe57b5bcc21eba2623b8a0a91 not found: ID does not exist" containerID="5a0e0ef3e1372450d7b3db0950bcf2e991ac9a5fe57b5bcc21eba2623b8a0a91"
Sep 08 13:52:50.323983 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:50.323953    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0e0ef3e1372450d7b3db0950bcf2e991ac9a5fe57b5bcc21eba2623b8a0a91"} err="failed to get container status \"5a0e0ef3e1372450d7b3db0950bcf2e991ac9a5fe57b5bcc21eba2623b8a0a91\": rpc error: code = NotFound desc = could not find container \"5a0e0ef3e1372450d7b3db0950bcf2e991ac9a5fe57b5bcc21eba2623b8a0a91\": container with ID starting with 5a0e0ef3e1372450d7b3db0950bcf2e991ac9a5fe57b5bcc21eba2623b8a0a91 not found: ID does not exist"
Sep 08 13:52:50.344737 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:50.344700    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7"]
Sep 08 13:52:50.349949 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:50.349922    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57856hvd7"]
Sep 08 13:52:51.006425 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:52:51.006392    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e16e6f4a-11ba-4d15-84cc-9f24923aa21d" path="/var/lib/kubelet/pods/e16e6f4a-11ba-4d15-84cc-9f24923aa21d/volumes"
Sep 08 13:53:37.665439 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:37.665363    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q"]
Sep 08 13:53:37.665900 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:37.665699    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e16e6f4a-11ba-4d15-84cc-9f24923aa21d" containerName="main"
Sep 08 13:53:37.665900 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:37.665711    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e16e6f4a-11ba-4d15-84cc-9f24923aa21d" containerName="main"
Sep 08 13:53:37.665900 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:37.665720    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f995836b-adf2-4ded-affb-0d91225898b5" containerName="main"
Sep 08 13:53:37.665900 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:37.665727    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f995836b-adf2-4ded-affb-0d91225898b5" containerName="main"
Sep 08 13:53:37.665900 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:37.665794    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e16e6f4a-11ba-4d15-84cc-9f24923aa21d" containerName="main"
Sep 08 13:53:37.665900 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:37.665803    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f995836b-adf2-4ded-affb-0d91225898b5" containerName="main"
Sep 08 13:53:37.668715 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:37.668691    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q"
Sep 08 13:53:37.673593 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:37.673564    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 08 13:53:37.673742 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:37.673604    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 08 13:53:37.674344 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:37.674321    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-bdrtb\""
Sep 08 13:53:37.674482 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:37.674459    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-pwm66\""
Sep 08 13:53:37.674589 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:37.674570    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 08 13:53:37.685543 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:37.685518    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q"]
Sep 08 13:53:37.787673 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:37.787636    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cdqq\" (UniqueName: \"kubernetes.io/projected/a8bd4af1-1a6a-4345-aee1-8169419dad81-kube-api-access-2cdqq\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q\" (UID: \"a8bd4af1-1a6a-4345-aee1-8169419dad81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q"
Sep 08 13:53:37.787872 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:37.787687    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bd4af1-1a6a-4345-aee1-8169419dad81-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q\" (UID: \"a8bd4af1-1a6a-4345-aee1-8169419dad81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q"
Sep 08 13:53:37.888127 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:37.888095    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cdqq\" (UniqueName: \"kubernetes.io/projected/a8bd4af1-1a6a-4345-aee1-8169419dad81-kube-api-access-2cdqq\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q\" (UID: \"a8bd4af1-1a6a-4345-aee1-8169419dad81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q"
Sep 08 13:53:37.888324 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:37.888143    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bd4af1-1a6a-4345-aee1-8169419dad81-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q\" (UID: \"a8bd4af1-1a6a-4345-aee1-8169419dad81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q"
Sep 08 13:53:37.890740 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:37.890717    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bd4af1-1a6a-4345-aee1-8169419dad81-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q\" (UID: \"a8bd4af1-1a6a-4345-aee1-8169419dad81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q"
Sep 08 13:53:37.896215 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:37.896192    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cdqq\" (UniqueName: \"kubernetes.io/projected/a8bd4af1-1a6a-4345-aee1-8169419dad81-kube-api-access-2cdqq\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q\" (UID: \"a8bd4af1-1a6a-4345-aee1-8169419dad81\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q"
Sep 08 13:53:37.981176 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:37.981145    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q"
Sep 08 13:53:38.258350 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:38.258320    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q"]
Sep 08 13:53:38.260090 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:53:38.260060    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bd4af1_1a6a_4345_aee1_8169419dad81.slice/crio-2f486e76785bef4e121c39fb05f85fdef138566c16f3f50bc57a16fe0bb673d6 WatchSource:0}: Error finding container 2f486e76785bef4e121c39fb05f85fdef138566c16f3f50bc57a16fe0bb673d6: Status 404 returned error can't find the container with id 2f486e76785bef4e121c39fb05f85fdef138566c16f3f50bc57a16fe0bb673d6
Sep 08 13:53:38.582974 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:38.582869    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q" event={"ID":"a8bd4af1-1a6a-4345-aee1-8169419dad81","Type":"ContainerStarted","Data":"f024b77e90e858ca403e643804feb4c3e6fb36250e902635ff0c2328f10d9f0e"}
Sep 08 13:53:38.582974 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:38.582933    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q" event={"ID":"a8bd4af1-1a6a-4345-aee1-8169419dad81","Type":"ContainerStarted","Data":"2f486e76785bef4e121c39fb05f85fdef138566c16f3f50bc57a16fe0bb673d6"}
Sep 08 13:53:38.583164 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:38.582977    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q"
Sep 08 13:53:38.600558 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:53:38.600461    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q" podStartSLOduration=1.600439616 podStartE2EDuration="1.600439616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:53:38.59918402 +0000 UTC m=+2250.059828845" watchObservedRunningTime="2026-09-08 13:53:38.600439616 +0000 UTC m=+2250.061084429"
Sep 08 13:54:09.590616 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:54:09.590586    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q"
Sep 08 13:54:14.570755 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:54:14.570716    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q"]
Sep 08 13:54:19.590721 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:54:19.590683    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q" podUID="a8bd4af1-1a6a-4345-aee1-8169419dad81" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:54:29.591042 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:54:29.591001    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q" podUID="a8bd4af1-1a6a-4345-aee1-8169419dad81" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:54:29.595806 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:54:29.595776    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q" podUID="a8bd4af1-1a6a-4345-aee1-8169419dad81" containerName="main" containerID="cri-o://f024b77e90e858ca403e643804feb4c3e6fb36250e902635ff0c2328f10d9f0e" gracePeriod=45
Sep 08 13:54:39.590860 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:54:39.590819    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q" podUID="a8bd4af1-1a6a-4345-aee1-8169419dad81" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:54:39.591694 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:54:39.590929    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q"
Sep 08 13:54:49.591703 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:54:49.591659    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q" podUID="a8bd4af1-1a6a-4345-aee1-8169419dad81" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:54:59.591268 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:54:59.591229    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q" podUID="a8bd4af1-1a6a-4345-aee1-8169419dad81" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:54:59.836119 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:54:59.836092    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q"
Sep 08 13:54:59.927569 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:54:59.927459    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bd4af1-1a6a-4345-aee1-8169419dad81-tls-certs\") pod \"a8bd4af1-1a6a-4345-aee1-8169419dad81\" (UID: \"a8bd4af1-1a6a-4345-aee1-8169419dad81\") "
Sep 08 13:54:59.927569 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:54:59.927548    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cdqq\" (UniqueName: \"kubernetes.io/projected/a8bd4af1-1a6a-4345-aee1-8169419dad81-kube-api-access-2cdqq\") pod \"a8bd4af1-1a6a-4345-aee1-8169419dad81\" (UID: \"a8bd4af1-1a6a-4345-aee1-8169419dad81\") "
Sep 08 13:54:59.929785 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:54:59.929756    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bd4af1-1a6a-4345-aee1-8169419dad81-kube-api-access-2cdqq" (OuterVolumeSpecName: "kube-api-access-2cdqq") pod "a8bd4af1-1a6a-4345-aee1-8169419dad81" (UID: "a8bd4af1-1a6a-4345-aee1-8169419dad81"). InnerVolumeSpecName "kube-api-access-2cdqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:54:59.929893 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:54:59.929802    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8bd4af1-1a6a-4345-aee1-8169419dad81-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8bd4af1-1a6a-4345-aee1-8169419dad81" (UID: "a8bd4af1-1a6a-4345-aee1-8169419dad81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:55:00.028709 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:00.028659    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bd4af1-1a6a-4345-aee1-8169419dad81-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:55:00.028709 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:00.028705    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cdqq\" (UniqueName: \"kubernetes.io/projected/a8bd4af1-1a6a-4345-aee1-8169419dad81-kube-api-access-2cdqq\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:55:00.036369 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:00.036334    2561 generic.go:356] "Generic (PLEG): container finished" podID="a8bd4af1-1a6a-4345-aee1-8169419dad81" containerID="f024b77e90e858ca403e643804feb4c3e6fb36250e902635ff0c2328f10d9f0e" exitCode=0
Sep 08 13:55:00.036510 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:00.036385    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q"
Sep 08 13:55:00.036510 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:00.036428    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q" event={"ID":"a8bd4af1-1a6a-4345-aee1-8169419dad81","Type":"ContainerDied","Data":"f024b77e90e858ca403e643804feb4c3e6fb36250e902635ff0c2328f10d9f0e"}
Sep 08 13:55:00.036510 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:00.036482    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q" event={"ID":"a8bd4af1-1a6a-4345-aee1-8169419dad81","Type":"ContainerDied","Data":"2f486e76785bef4e121c39fb05f85fdef138566c16f3f50bc57a16fe0bb673d6"}
Sep 08 13:55:00.036510 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:00.036507    2561 scope.go:117] "RemoveContainer" containerID="f024b77e90e858ca403e643804feb4c3e6fb36250e902635ff0c2328f10d9f0e"
Sep 08 13:55:00.046610 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:00.046592    2561 scope.go:117] "RemoveContainer" containerID="f024b77e90e858ca403e643804feb4c3e6fb36250e902635ff0c2328f10d9f0e"
Sep 08 13:55:00.046942 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:55:00.046920    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f024b77e90e858ca403e643804feb4c3e6fb36250e902635ff0c2328f10d9f0e\": container with ID starting with f024b77e90e858ca403e643804feb4c3e6fb36250e902635ff0c2328f10d9f0e not found: ID does not exist" containerID="f024b77e90e858ca403e643804feb4c3e6fb36250e902635ff0c2328f10d9f0e"
Sep 08 13:55:00.047011 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:00.046958    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f024b77e90e858ca403e643804feb4c3e6fb36250e902635ff0c2328f10d9f0e"} err="failed to get container status \"f024b77e90e858ca403e643804feb4c3e6fb36250e902635ff0c2328f10d9f0e\": rpc error: code = NotFound desc = could not find container \"f024b77e90e858ca403e643804feb4c3e6fb36250e902635ff0c2328f10d9f0e\": container with ID starting with f024b77e90e858ca403e643804feb4c3e6fb36250e902635ff0c2328f10d9f0e not found: ID does not exist"
Sep 08 13:55:00.061747 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:00.061715    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q"]
Sep 08 13:55:00.069055 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:00.069028    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dndg9q"]
Sep 08 13:55:01.006998 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:01.006962    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8bd4af1-1a6a-4345-aee1-8169419dad81" path="/var/lib/kubelet/pods/a8bd4af1-1a6a-4345-aee1-8169419dad81/volumes"
Sep 08 13:55:16.749520 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:16.749488    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml"]
Sep 08 13:55:16.750038 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:16.749876    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8bd4af1-1a6a-4345-aee1-8169419dad81" containerName="main"
Sep 08 13:55:16.750038 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:16.749890    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8bd4af1-1a6a-4345-aee1-8169419dad81" containerName="main"
Sep 08 13:55:16.750038 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:16.749952    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8bd4af1-1a6a-4345-aee1-8169419dad81" containerName="main"
Sep 08 13:55:16.752980 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:16.752963    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml"
Sep 08 13:55:16.755759 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:16.755733    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 08 13:55:16.755895 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:16.755779    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 08 13:55:16.756724 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:16.756702    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-qh8m9\""
Sep 08 13:55:16.756852 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:16.756837    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 08 13:55:16.756986 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:16.756972    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-8dkz9\""
Sep 08 13:55:16.763753 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:16.763728    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml"]
Sep 08 13:55:16.878481 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:16.878415    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbbfdc00-2ba2-4b26-bf03-74506fc64cd9-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml\" (UID: \"cbbfdc00-2ba2-4b26-bf03-74506fc64cd9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml"
Sep 08 13:55:16.878668 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:16.878582    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxb49\" (UniqueName: \"kubernetes.io/projected/cbbfdc00-2ba2-4b26-bf03-74506fc64cd9-kube-api-access-qxb49\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml\" (UID: \"cbbfdc00-2ba2-4b26-bf03-74506fc64cd9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml"
Sep 08 13:55:16.979498 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:16.979460    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxb49\" (UniqueName: \"kubernetes.io/projected/cbbfdc00-2ba2-4b26-bf03-74506fc64cd9-kube-api-access-qxb49\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml\" (UID: \"cbbfdc00-2ba2-4b26-bf03-74506fc64cd9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml"
Sep 08 13:55:16.979668 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:16.979531    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbbfdc00-2ba2-4b26-bf03-74506fc64cd9-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml\" (UID: \"cbbfdc00-2ba2-4b26-bf03-74506fc64cd9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml"
Sep 08 13:55:16.981996 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:16.981967    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbbfdc00-2ba2-4b26-bf03-74506fc64cd9-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml\" (UID: \"cbbfdc00-2ba2-4b26-bf03-74506fc64cd9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml"
Sep 08 13:55:16.989723 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:16.989698    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxb49\" (UniqueName: \"kubernetes.io/projected/cbbfdc00-2ba2-4b26-bf03-74506fc64cd9-kube-api-access-qxb49\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml\" (UID: \"cbbfdc00-2ba2-4b26-bf03-74506fc64cd9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml"
Sep 08 13:55:17.064204 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:17.064110    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml"
Sep 08 13:55:17.336698 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:17.336669    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml"]
Sep 08 13:55:17.338710 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:55:17.338681    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbbfdc00_2ba2_4b26_bf03_74506fc64cd9.slice/crio-2a2244ffd000e5e7895c8cb68f7c51c03ba0b895a10f20abc95e71dc6418d775 WatchSource:0}: Error finding container 2a2244ffd000e5e7895c8cb68f7c51c03ba0b895a10f20abc95e71dc6418d775: Status 404 returned error can't find the container with id 2a2244ffd000e5e7895c8cb68f7c51c03ba0b895a10f20abc95e71dc6418d775
Sep 08 13:55:18.136108 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:18.136061    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml" event={"ID":"cbbfdc00-2ba2-4b26-bf03-74506fc64cd9","Type":"ContainerStarted","Data":"6f454b8f3c054e0d7ce201bde1db7097fccc0dbadda8e126b102ebdb9427742e"}
Sep 08 13:55:18.136625 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:18.136127    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml" event={"ID":"cbbfdc00-2ba2-4b26-bf03-74506fc64cd9","Type":"ContainerStarted","Data":"2a2244ffd000e5e7895c8cb68f7c51c03ba0b895a10f20abc95e71dc6418d775"}
Sep 08 13:55:18.136625 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:18.136176    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml"
Sep 08 13:55:18.159314 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:18.159256    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml" podStartSLOduration=2.159240001 podStartE2EDuration="2.159240001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:55:18.156926716 +0000 UTC m=+2349.617571527" watchObservedRunningTime="2026-09-08 13:55:18.159240001 +0000 UTC m=+2349.619884823"
Sep 08 13:55:49.143880 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:55:49.143849    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml"
Sep 08 13:56:04.069240 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:04.069189    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml"]
Sep 08 13:56:09.143310 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:09.143251    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml" podUID="cbbfdc00-2ba2-4b26-bf03-74506fc64cd9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:56:09.166497 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:09.166471    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 13:56:09.178986 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:09.178961    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 13:56:19.090061 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:19.089995    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml" podUID="cbbfdc00-2ba2-4b26-bf03-74506fc64cd9" containerName="main" containerID="cri-o://6f454b8f3c054e0d7ce201bde1db7097fccc0dbadda8e126b102ebdb9427742e" gracePeriod=45
Sep 08 13:56:19.143770 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:19.143731    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml" podUID="cbbfdc00-2ba2-4b26-bf03-74506fc64cd9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:56:29.143553 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:29.143513    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml" podUID="cbbfdc00-2ba2-4b26-bf03-74506fc64cd9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:56:29.144002 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:29.143630    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml"
Sep 08 13:56:39.144223 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:39.144136    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml" podUID="cbbfdc00-2ba2-4b26-bf03-74506fc64cd9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:56:49.142937 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:56:49.142900    2561 logging.go:55] [core] [Channel #596 SubChannel #597]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.65:9003", ServerName: "10.134.0.65:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.65:9003: connect: connection refused"
Sep 08 13:56:49.336717 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:49.336688    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml"
Sep 08 13:56:49.489988 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:49.489952    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbbfdc00-2ba2-4b26-bf03-74506fc64cd9-tls-certs\") pod \"cbbfdc00-2ba2-4b26-bf03-74506fc64cd9\" (UID: \"cbbfdc00-2ba2-4b26-bf03-74506fc64cd9\") "
Sep 08 13:56:49.490160 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:49.490079    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxb49\" (UniqueName: \"kubernetes.io/projected/cbbfdc00-2ba2-4b26-bf03-74506fc64cd9-kube-api-access-qxb49\") pod \"cbbfdc00-2ba2-4b26-bf03-74506fc64cd9\" (UID: \"cbbfdc00-2ba2-4b26-bf03-74506fc64cd9\") "
Sep 08 13:56:49.492301 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:49.492250    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbbfdc00-2ba2-4b26-bf03-74506fc64cd9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cbbfdc00-2ba2-4b26-bf03-74506fc64cd9" (UID: "cbbfdc00-2ba2-4b26-bf03-74506fc64cd9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:56:49.492301 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:49.492267    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbbfdc00-2ba2-4b26-bf03-74506fc64cd9-kube-api-access-qxb49" (OuterVolumeSpecName: "kube-api-access-qxb49") pod "cbbfdc00-2ba2-4b26-bf03-74506fc64cd9" (UID: "cbbfdc00-2ba2-4b26-bf03-74506fc64cd9"). InnerVolumeSpecName "kube-api-access-qxb49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:56:49.590938 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:49.590902    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxb49\" (UniqueName: \"kubernetes.io/projected/cbbfdc00-2ba2-4b26-bf03-74506fc64cd9-kube-api-access-qxb49\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:56:49.590938 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:49.590932    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbbfdc00-2ba2-4b26-bf03-74506fc64cd9-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:56:49.616735 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:49.616694    2561 generic.go:356] "Generic (PLEG): container finished" podID="cbbfdc00-2ba2-4b26-bf03-74506fc64cd9" containerID="6f454b8f3c054e0d7ce201bde1db7097fccc0dbadda8e126b102ebdb9427742e" exitCode=0
Sep 08 13:56:49.616914 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:49.616744    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml"
Sep 08 13:56:49.616914 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:49.616780    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml" event={"ID":"cbbfdc00-2ba2-4b26-bf03-74506fc64cd9","Type":"ContainerDied","Data":"6f454b8f3c054e0d7ce201bde1db7097fccc0dbadda8e126b102ebdb9427742e"}
Sep 08 13:56:49.616914 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:49.616846    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml" event={"ID":"cbbfdc00-2ba2-4b26-bf03-74506fc64cd9","Type":"ContainerDied","Data":"2a2244ffd000e5e7895c8cb68f7c51c03ba0b895a10f20abc95e71dc6418d775"}
Sep 08 13:56:49.616914 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:49.616879    2561 scope.go:117] "RemoveContainer" containerID="6f454b8f3c054e0d7ce201bde1db7097fccc0dbadda8e126b102ebdb9427742e"
Sep 08 13:56:49.625970 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:49.625951    2561 scope.go:117] "RemoveContainer" containerID="6f454b8f3c054e0d7ce201bde1db7097fccc0dbadda8e126b102ebdb9427742e"
Sep 08 13:56:49.626258 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:56:49.626238    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f454b8f3c054e0d7ce201bde1db7097fccc0dbadda8e126b102ebdb9427742e\": container with ID starting with 6f454b8f3c054e0d7ce201bde1db7097fccc0dbadda8e126b102ebdb9427742e not found: ID does not exist" containerID="6f454b8f3c054e0d7ce201bde1db7097fccc0dbadda8e126b102ebdb9427742e"
Sep 08 13:56:49.626394 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:49.626266    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f454b8f3c054e0d7ce201bde1db7097fccc0dbadda8e126b102ebdb9427742e"} err="failed to get container status \"6f454b8f3c054e0d7ce201bde1db7097fccc0dbadda8e126b102ebdb9427742e\": rpc error: code = NotFound desc = could not find container \"6f454b8f3c054e0d7ce201bde1db7097fccc0dbadda8e126b102ebdb9427742e\": container with ID starting with 6f454b8f3c054e0d7ce201bde1db7097fccc0dbadda8e126b102ebdb9427742e not found: ID does not exist"
Sep 08 13:56:49.639698 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:49.639671    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml"]
Sep 08 13:56:49.645852 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:49.645827    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml"]
Sep 08 13:56:50.143043 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:50.142989    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebhmml" podUID="cbbfdc00-2ba2-4b26-bf03-74506fc64cd9" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.65:9003\" within 1s: context deadline exceeded"
Sep 08 13:56:51.007070 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:51.007032    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbbfdc00-2ba2-4b26-bf03-74506fc64cd9" path="/var/lib/kubelet/pods/cbbfdc00-2ba2-4b26-bf03-74506fc64cd9/volumes"
Sep 08 13:56:52.865035 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:52.864994    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"]
Sep 08 13:56:52.865468 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:52.865365    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbbfdc00-2ba2-4b26-bf03-74506fc64cd9" containerName="main"
Sep 08 13:56:52.865468 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:52.865377    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbbfdc00-2ba2-4b26-bf03-74506fc64cd9" containerName="main"
Sep 08 13:56:52.865468 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:52.865445    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbbfdc00-2ba2-4b26-bf03-74506fc64cd9" containerName="main"
Sep 08 13:56:52.869912 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:52.869895    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:56:52.873451 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:52.873422    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 08 13:56:52.874312 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:52.874273    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v2-kserve-self-signed-certs\""
Sep 08 13:56:52.874442 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:52.874315    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 08 13:56:52.877852 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:52.877829    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-nksjg\""
Sep 08 13:56:52.885185 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:52.885157    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"]
Sep 08 13:56:53.019816 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.019775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-wvjsv\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:56:53.019998 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.019836    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-wvjsv\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:56:53.019998 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.019893    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jqz6\" (UniqueName: \"kubernetes.io/projected/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-kube-api-access-7jqz6\") pod \"delw-v2-kserve-5b84cc7ddf-wvjsv\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:56:53.019998 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.019962    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-home\") pod \"delw-v2-kserve-5b84cc7ddf-wvjsv\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:56:53.019998 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.019989    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-wvjsv\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:56:53.020201 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.020094    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-wvjsv\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:56:53.121080 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.120981    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-home\") pod \"delw-v2-kserve-5b84cc7ddf-wvjsv\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:56:53.121080 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.121028    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-wvjsv\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:56:53.121334 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.121086    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-wvjsv\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:56:53.121334 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.121119    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-wvjsv\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:56:53.121334 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.121241    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-wvjsv\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:56:53.121334 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.121327    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jqz6\" (UniqueName: \"kubernetes.io/projected/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-kube-api-access-7jqz6\") pod \"delw-v2-kserve-5b84cc7ddf-wvjsv\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:56:53.121563 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.121402    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-home\") pod \"delw-v2-kserve-5b84cc7ddf-wvjsv\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:56:53.121563 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.121466    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-wvjsv\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:56:53.121563 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.121515    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-wvjsv\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:56:53.123395 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.123376    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-wvjsv\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:56:53.123690 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.123671    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-wvjsv\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:56:53.130902 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.130877    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jqz6\" (UniqueName: \"kubernetes.io/projected/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-kube-api-access-7jqz6\") pod \"delw-v2-kserve-5b84cc7ddf-wvjsv\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:56:53.180295 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.180242    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:56:53.464414 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.464383    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"]
Sep 08 13:56:53.465985 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:56:53.465960    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3acda7ac_ea5e_47b8_9fd9_3f83ef7a57aa.slice/crio-0b5a173ee2cca7ae5d0c5b671f4b2b69770d24e976b56ea0f28036d93e7b60c2 WatchSource:0}: Error finding container 0b5a173ee2cca7ae5d0c5b671f4b2b69770d24e976b56ea0f28036d93e7b60c2: Status 404 returned error can't find the container with id 0b5a173ee2cca7ae5d0c5b671f4b2b69770d24e976b56ea0f28036d93e7b60c2
Sep 08 13:56:53.643482 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.643443    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv" event={"ID":"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa","Type":"ContainerStarted","Data":"a479f9d186be0cfa6fcc47c58cba7bab19203d229e941f6ad8c6cb8bc6adffc4"}
Sep 08 13:56:53.643670 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.643511    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv" event={"ID":"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa","Type":"ContainerStarted","Data":"0b5a173ee2cca7ae5d0c5b671f4b2b69770d24e976b56ea0f28036d93e7b60c2"}
Sep 08 13:56:53.668746 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:53.668684    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv" podStartSLOduration=1.668665516 podStartE2EDuration="1.668665516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:56:53.66662948 +0000 UTC m=+2445.127274290" watchObservedRunningTime="2026-09-08 13:56:53.668665516 +0000 UTC m=+2445.129310325"
Sep 08 13:56:54.181027 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:56:54.180991    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:57:03.180729 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:03.180686    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:57:03.193595 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:03.193568    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:57:03.713454 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:03.713422    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:57:05.907809 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:05.907772    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc"]
Sep 08 13:57:05.911552 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:05.911530    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc"
Sep 08 13:57:05.914042 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:05.914020    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 08 13:57:05.914781 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:05.914760    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-l7glm\""
Sep 08 13:57:05.915111 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:05.915092    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-zq4fq\""
Sep 08 13:57:05.915488 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:05.915470    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 08 13:57:05.915724 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:05.915709    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 08 13:57:05.923777 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:05.923752    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc"]
Sep 08 13:57:06.040407 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:06.040365    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4205b4e6-501d-4aae-9020-3153fdc26b53-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc\" (UID: \"4205b4e6-501d-4aae-9020-3153fdc26b53\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc"
Sep 08 13:57:06.040621 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:06.040462    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn6dk\" (UniqueName: \"kubernetes.io/projected/4205b4e6-501d-4aae-9020-3153fdc26b53-kube-api-access-dn6dk\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc\" (UID: \"4205b4e6-501d-4aae-9020-3153fdc26b53\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc"
Sep 08 13:57:06.141374 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:06.141189    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4205b4e6-501d-4aae-9020-3153fdc26b53-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc\" (UID: \"4205b4e6-501d-4aae-9020-3153fdc26b53\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc"
Sep 08 13:57:06.142000 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:06.141571    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn6dk\" (UniqueName: \"kubernetes.io/projected/4205b4e6-501d-4aae-9020-3153fdc26b53-kube-api-access-dn6dk\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc\" (UID: \"4205b4e6-501d-4aae-9020-3153fdc26b53\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc"
Sep 08 13:57:06.144532 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:06.144503    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4205b4e6-501d-4aae-9020-3153fdc26b53-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc\" (UID: \"4205b4e6-501d-4aae-9020-3153fdc26b53\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc"
Sep 08 13:57:06.151969 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:06.151940    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn6dk\" (UniqueName: \"kubernetes.io/projected/4205b4e6-501d-4aae-9020-3153fdc26b53-kube-api-access-dn6dk\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc\" (UID: \"4205b4e6-501d-4aae-9020-3153fdc26b53\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc"
Sep 08 13:57:06.222782 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:06.222741    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc"
Sep 08 13:57:06.501565 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:06.501538    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc"]
Sep 08 13:57:06.503716 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:57:06.503686    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4205b4e6_501d_4aae_9020_3153fdc26b53.slice/crio-d1e69aaf47c913cfc23acc512c72e6599a6616839fc4ad19208c60423bd84b29 WatchSource:0}: Error finding container d1e69aaf47c913cfc23acc512c72e6599a6616839fc4ad19208c60423bd84b29: Status 404 returned error can't find the container with id d1e69aaf47c913cfc23acc512c72e6599a6616839fc4ad19208c60423bd84b29
Sep 08 13:57:06.719964 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:06.719930    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc" event={"ID":"4205b4e6-501d-4aae-9020-3153fdc26b53","Type":"ContainerStarted","Data":"2b3062de61e95c0d3af6132c3ed9f178adc27587a00ff3ae4e27a1a87ba896a4"}
Sep 08 13:57:06.720193 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:06.719990    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc" event={"ID":"4205b4e6-501d-4aae-9020-3153fdc26b53","Type":"ContainerStarted","Data":"d1e69aaf47c913cfc23acc512c72e6599a6616839fc4ad19208c60423bd84b29"}
Sep 08 13:57:06.720193 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:06.720019    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc"
Sep 08 13:57:06.743171 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:06.743108    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc" podStartSLOduration=1.743084313 podStartE2EDuration="1.743084313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:57:06.740718244 +0000 UTC m=+2458.201363056" watchObservedRunningTime="2026-09-08 13:57:06.743084313 +0000 UTC m=+2458.203729126"
Sep 08 13:57:37.726894 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:37.726862    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc"
Sep 08 13:57:44.357252 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:44.357214    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk"]
Sep 08 13:57:44.361188 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:44.361165    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk"
Sep 08 13:57:44.364792 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:44.364764    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-4r4jv\""
Sep 08 13:57:44.364792 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:44.364785    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 08 13:57:44.372881 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:44.372856    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk"]
Sep 08 13:57:44.519507 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:44.519455    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk\" (UID: \"8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk"
Sep 08 13:57:44.519707 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:44.519611    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9jlz\" (UniqueName: \"kubernetes.io/projected/8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe-kube-api-access-c9jlz\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk\" (UID: \"8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk"
Sep 08 13:57:44.620883 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:44.620768    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9jlz\" (UniqueName: \"kubernetes.io/projected/8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe-kube-api-access-c9jlz\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk\" (UID: \"8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk"
Sep 08 13:57:44.620883 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:44.620846    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk\" (UID: \"8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk"
Sep 08 13:57:44.623770 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:44.623739    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk\" (UID: \"8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk"
Sep 08 13:57:44.630177 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:44.630145    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9jlz\" (UniqueName: \"kubernetes.io/projected/8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe-kube-api-access-c9jlz\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk\" (UID: \"8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk"
Sep 08 13:57:44.673479 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:44.673439    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk"
Sep 08 13:57:44.957798 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:44.957771    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk"]
Sep 08 13:57:44.959919 ip-10-0-130-178 kubenswrapper[2561]: W0908 13:57:44.959889    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e06b8ed_7757_4b07_a7e3_0b0f1e561ebe.slice/crio-df2b072faa4f50cc9e9f6121428ef6144f33cf1af5f154c25f0707c01467d794 WatchSource:0}: Error finding container df2b072faa4f50cc9e9f6121428ef6144f33cf1af5f154c25f0707c01467d794: Status 404 returned error can't find the container with id df2b072faa4f50cc9e9f6121428ef6144f33cf1af5f154c25f0707c01467d794
Sep 08 13:57:45.936243 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:45.936208    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk" event={"ID":"8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe","Type":"ContainerStarted","Data":"4ffd60826b945064b782ce75694d4c8de53a5cd68a685c87d3e873c37ff1a6ae"}
Sep 08 13:57:45.936648 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:45.936270    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk" event={"ID":"8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe","Type":"ContainerStarted","Data":"df2b072faa4f50cc9e9f6121428ef6144f33cf1af5f154c25f0707c01467d794"}
Sep 08 13:57:45.936648 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:45.936396    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk"
Sep 08 13:57:48.501782 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:48.499498    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk" podStartSLOduration=4.499474124 podStartE2EDuration="4.499474124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:57:45.982707016 +0000 UTC m=+2497.443351827" watchObservedRunningTime="2026-09-08 13:57:48.499474124 +0000 UTC m=+2499.960118935"
Sep 08 13:57:48.501782 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:57:48.500522    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"]
Sep 08 13:58:03.522657 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:03.522548    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv" podUID="3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa" containerName="main" containerID="cri-o://a479f9d186be0cfa6fcc47c58cba7bab19203d229e941f6ad8c6cb8bc6adffc4" gracePeriod=45
Sep 08 13:58:03.775586 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:03.775507    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:58:03.895441 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:03.895406    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-home\") pod \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") "
Sep 08 13:58:03.895647 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:03.895464    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-dshm\") pod \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") "
Sep 08 13:58:03.895647 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:03.895500    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-tls-certs\") pod \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") "
Sep 08 13:58:03.895647 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:03.895549    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jqz6\" (UniqueName: \"kubernetes.io/projected/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-kube-api-access-7jqz6\") pod \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") "
Sep 08 13:58:03.895647 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:03.895604    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-tmp-dir\") pod \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") "
Sep 08 13:58:03.895938 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:03.895655    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-model-cache\") pod \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\" (UID: \"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa\") "
Sep 08 13:58:03.895938 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:03.895736    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-home" (OuterVolumeSpecName: "home") pod "3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa" (UID: "3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:58:03.896053 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:03.895947    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa" (UID: "3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:58:03.896053 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:03.895981    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa" (UID: "3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:58:03.896053 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:03.896027    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-home\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:58:03.897859 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:03.897835    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-kube-api-access-7jqz6" (OuterVolumeSpecName: "kube-api-access-7jqz6") pod "3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa" (UID: "3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa"). InnerVolumeSpecName "kube-api-access-7jqz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:58:03.898128 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:03.898108    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-dshm" (OuterVolumeSpecName: "dshm") pod "3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa" (UID: "3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:58:03.898128 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:03.898118    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa" (UID: "3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:58:03.997028 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:03.996977    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:58:03.997028 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:03.997022    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jqz6\" (UniqueName: \"kubernetes.io/projected/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-kube-api-access-7jqz6\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:58:03.997028 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:03.997032    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-tmp-dir\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:58:03.997028 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:03.997043    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-model-cache\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:58:03.997028 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:03.997051    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa-dshm\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:58:04.039204 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:04.039114    2561 generic.go:356] "Generic (PLEG): container finished" podID="3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa" containerID="a479f9d186be0cfa6fcc47c58cba7bab19203d229e941f6ad8c6cb8bc6adffc4" exitCode=0
Sep 08 13:58:04.039204 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:04.039159    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"
Sep 08 13:58:04.039430 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:04.039203    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv" event={"ID":"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa","Type":"ContainerDied","Data":"a479f9d186be0cfa6fcc47c58cba7bab19203d229e941f6ad8c6cb8bc6adffc4"}
Sep 08 13:58:04.039430 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:04.039259    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv" event={"ID":"3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa","Type":"ContainerDied","Data":"0b5a173ee2cca7ae5d0c5b671f4b2b69770d24e976b56ea0f28036d93e7b60c2"}
Sep 08 13:58:04.039430 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:04.039283    2561 scope.go:117] "RemoveContainer" containerID="a479f9d186be0cfa6fcc47c58cba7bab19203d229e941f6ad8c6cb8bc6adffc4"
Sep 08 13:58:04.049139 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:04.049120    2561 scope.go:117] "RemoveContainer" containerID="a479f9d186be0cfa6fcc47c58cba7bab19203d229e941f6ad8c6cb8bc6adffc4"
Sep 08 13:58:04.049435 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:58:04.049414    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a479f9d186be0cfa6fcc47c58cba7bab19203d229e941f6ad8c6cb8bc6adffc4\": container with ID starting with a479f9d186be0cfa6fcc47c58cba7bab19203d229e941f6ad8c6cb8bc6adffc4 not found: ID does not exist" containerID="a479f9d186be0cfa6fcc47c58cba7bab19203d229e941f6ad8c6cb8bc6adffc4"
Sep 08 13:58:04.049488 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:04.049444    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a479f9d186be0cfa6fcc47c58cba7bab19203d229e941f6ad8c6cb8bc6adffc4"} err="failed to get container status \"a479f9d186be0cfa6fcc47c58cba7bab19203d229e941f6ad8c6cb8bc6adffc4\": rpc error: code = NotFound desc = could not find container \"a479f9d186be0cfa6fcc47c58cba7bab19203d229e941f6ad8c6cb8bc6adffc4\": container with ID starting with a479f9d186be0cfa6fcc47c58cba7bab19203d229e941f6ad8c6cb8bc6adffc4 not found: ID does not exist"
Sep 08 13:58:04.071256 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:04.071223    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"]
Sep 08 13:58:04.073354 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:04.073333    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv"]
Sep 08 13:58:04.703009 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:04.702949    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-wvjsv" podUID="3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 08 13:58:05.008044 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:05.007957    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa" path="/var/lib/kubelet/pods/3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa/volumes"
Sep 08 13:58:16.944796 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:58:16.944760    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk"
Sep 08 13:59:05.154106 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:05.154057    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk"]
Sep 08 13:59:06.944972 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:06.944929    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk" podUID="8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:59:16.945592 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:16.945552    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk" podUID="8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:59:20.186614 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:20.186539    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk" podUID="8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe" containerName="main" containerID="cri-o://4ffd60826b945064b782ce75694d4c8de53a5cd68a685c87d3e873c37ff1a6ae" gracePeriod=45
Sep 08 13:59:26.945787 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:26.945744    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk" podUID="8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:59:26.946191 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:26.945886    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk"
Sep 08 13:59:36.945204 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:36.945116    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk" podUID="8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:59:46.945120 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:46.945072    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk" podUID="8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:59:50.447784 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:50.447720    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk"
Sep 08 13:59:50.580023 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:50.579985    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9jlz\" (UniqueName: \"kubernetes.io/projected/8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe-kube-api-access-c9jlz\") pod \"8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe\" (UID: \"8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe\") "
Sep 08 13:59:50.580219 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:50.580050    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe-tls-certs\") pod \"8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe\" (UID: \"8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe\") "
Sep 08 13:59:50.582270 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:50.582236    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe" (UID: "8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:59:50.582427 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:50.582279    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe-kube-api-access-c9jlz" (OuterVolumeSpecName: "kube-api-access-c9jlz") pod "8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe" (UID: "8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe"). InnerVolumeSpecName "kube-api-access-c9jlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:59:50.614144 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:50.614111    2561 generic.go:356] "Generic (PLEG): container finished" podID="8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe" containerID="4ffd60826b945064b782ce75694d4c8de53a5cd68a685c87d3e873c37ff1a6ae" exitCode=0
Sep 08 13:59:50.614323 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:50.614156    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk"
Sep 08 13:59:50.614323 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:50.614182    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk" event={"ID":"8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe","Type":"ContainerDied","Data":"4ffd60826b945064b782ce75694d4c8de53a5cd68a685c87d3e873c37ff1a6ae"}
Sep 08 13:59:50.614323 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:50.614232    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk" event={"ID":"8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe","Type":"ContainerDied","Data":"df2b072faa4f50cc9e9f6121428ef6144f33cf1af5f154c25f0707c01467d794"}
Sep 08 13:59:50.614323 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:50.614259    2561 scope.go:117] "RemoveContainer" containerID="4ffd60826b945064b782ce75694d4c8de53a5cd68a685c87d3e873c37ff1a6ae"
Sep 08 13:59:50.625613 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:50.625457    2561 scope.go:117] "RemoveContainer" containerID="4ffd60826b945064b782ce75694d4c8de53a5cd68a685c87d3e873c37ff1a6ae"
Sep 08 13:59:50.626066 ip-10-0-130-178 kubenswrapper[2561]: E0908 13:59:50.626037    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ffd60826b945064b782ce75694d4c8de53a5cd68a685c87d3e873c37ff1a6ae\": container with ID starting with 4ffd60826b945064b782ce75694d4c8de53a5cd68a685c87d3e873c37ff1a6ae not found: ID does not exist" containerID="4ffd60826b945064b782ce75694d4c8de53a5cd68a685c87d3e873c37ff1a6ae"
Sep 08 13:59:50.626160 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:50.626076    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ffd60826b945064b782ce75694d4c8de53a5cd68a685c87d3e873c37ff1a6ae"} err="failed to get container status \"4ffd60826b945064b782ce75694d4c8de53a5cd68a685c87d3e873c37ff1a6ae\": rpc error: code = NotFound desc = could not find container \"4ffd60826b945064b782ce75694d4c8de53a5cd68a685c87d3e873c37ff1a6ae\": container with ID starting with 4ffd60826b945064b782ce75694d4c8de53a5cd68a685c87d3e873c37ff1a6ae not found: ID does not exist"
Sep 08 13:59:50.640598 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:50.640565    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk"]
Sep 08 13:59:50.644922 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:50.644888    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche598sk"]
Sep 08 13:59:50.681495 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:50.681454    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9jlz\" (UniqueName: \"kubernetes.io/projected/8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe-kube-api-access-c9jlz\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:59:50.681495 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:50.681490    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 13:59:51.007109 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:51.007075    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe" path="/var/lib/kubelet/pods/8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe/volumes"
Sep 08 13:59:52.872605 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:52.872568    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc"]
Sep 08 13:59:57.727512 ip-10-0-130-178 kubenswrapper[2561]: I0908 13:59:57.727469    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc" podUID="4205b4e6-501d-4aae-9020-3153fdc26b53" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:00:07.726655 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:07.726617    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc" podUID="4205b4e6-501d-4aae-9020-3153fdc26b53" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:00:07.892656 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:07.892617    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc" podUID="4205b4e6-501d-4aae-9020-3153fdc26b53" containerName="main" containerID="cri-o://2b3062de61e95c0d3af6132c3ed9f178adc27587a00ff3ae4e27a1a87ba896a4" gracePeriod=45
Sep 08 14:00:17.726528 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:17.726485    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc" podUID="4205b4e6-501d-4aae-9020-3153fdc26b53" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:00:17.726957 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:17.726597    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc"
Sep 08 14:00:27.726825 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:27.726783    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc" podUID="4205b4e6-501d-4aae-9020-3153fdc26b53" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:00:37.727272 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:37.727230    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc" podUID="4205b4e6-501d-4aae-9020-3153fdc26b53" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:00:38.148660 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:38.148632    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc"
Sep 08 14:00:38.216363 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:38.216324    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4205b4e6-501d-4aae-9020-3153fdc26b53-tls-certs\") pod \"4205b4e6-501d-4aae-9020-3153fdc26b53\" (UID: \"4205b4e6-501d-4aae-9020-3153fdc26b53\") "
Sep 08 14:00:38.216557 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:38.216372    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dn6dk\" (UniqueName: \"kubernetes.io/projected/4205b4e6-501d-4aae-9020-3153fdc26b53-kube-api-access-dn6dk\") pod \"4205b4e6-501d-4aae-9020-3153fdc26b53\" (UID: \"4205b4e6-501d-4aae-9020-3153fdc26b53\") "
Sep 08 14:00:38.218527 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:38.218490    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4205b4e6-501d-4aae-9020-3153fdc26b53-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4205b4e6-501d-4aae-9020-3153fdc26b53" (UID: "4205b4e6-501d-4aae-9020-3153fdc26b53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:00:38.218667 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:38.218572    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4205b4e6-501d-4aae-9020-3153fdc26b53-kube-api-access-dn6dk" (OuterVolumeSpecName: "kube-api-access-dn6dk") pod "4205b4e6-501d-4aae-9020-3153fdc26b53" (UID: "4205b4e6-501d-4aae-9020-3153fdc26b53"). InnerVolumeSpecName "kube-api-access-dn6dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:00:38.318113 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:38.318019    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4205b4e6-501d-4aae-9020-3153fdc26b53-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:00:38.318113 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:38.318050    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn6dk\" (UniqueName: \"kubernetes.io/projected/4205b4e6-501d-4aae-9020-3153fdc26b53-kube-api-access-dn6dk\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:00:38.879952 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:38.879916    2561 generic.go:356] "Generic (PLEG): container finished" podID="4205b4e6-501d-4aae-9020-3153fdc26b53" containerID="2b3062de61e95c0d3af6132c3ed9f178adc27587a00ff3ae4e27a1a87ba896a4" exitCode=0
Sep 08 14:00:38.880497 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:38.879961    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc"
Sep 08 14:00:38.880497 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:38.879962    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc" event={"ID":"4205b4e6-501d-4aae-9020-3153fdc26b53","Type":"ContainerDied","Data":"2b3062de61e95c0d3af6132c3ed9f178adc27587a00ff3ae4e27a1a87ba896a4"}
Sep 08 14:00:38.880497 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:38.880091    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc" event={"ID":"4205b4e6-501d-4aae-9020-3153fdc26b53","Type":"ContainerDied","Data":"d1e69aaf47c913cfc23acc512c72e6599a6616839fc4ad19208c60423bd84b29"}
Sep 08 14:00:38.880497 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:38.880120    2561 scope.go:117] "RemoveContainer" containerID="2b3062de61e95c0d3af6132c3ed9f178adc27587a00ff3ae4e27a1a87ba896a4"
Sep 08 14:00:38.890124 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:38.890100    2561 scope.go:117] "RemoveContainer" containerID="2b3062de61e95c0d3af6132c3ed9f178adc27587a00ff3ae4e27a1a87ba896a4"
Sep 08 14:00:38.890429 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:00:38.890409    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3062de61e95c0d3af6132c3ed9f178adc27587a00ff3ae4e27a1a87ba896a4\": container with ID starting with 2b3062de61e95c0d3af6132c3ed9f178adc27587a00ff3ae4e27a1a87ba896a4 not found: ID does not exist" containerID="2b3062de61e95c0d3af6132c3ed9f178adc27587a00ff3ae4e27a1a87ba896a4"
Sep 08 14:00:38.890496 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:38.890439    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3062de61e95c0d3af6132c3ed9f178adc27587a00ff3ae4e27a1a87ba896a4"} err="failed to get container status \"2b3062de61e95c0d3af6132c3ed9f178adc27587a00ff3ae4e27a1a87ba896a4\": rpc error: code = NotFound desc = could not find container \"2b3062de61e95c0d3af6132c3ed9f178adc27587a00ff3ae4e27a1a87ba896a4\": container with ID starting with 2b3062de61e95c0d3af6132c3ed9f178adc27587a00ff3ae4e27a1a87ba896a4 not found: ID does not exist"
Sep 08 14:00:38.901251 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:38.901223    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc"]
Sep 08 14:00:38.905115 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:38.905089    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schecbwkc"]
Sep 08 14:00:39.006921 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:00:39.006887    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4205b4e6-501d-4aae-9020-3153fdc26b53" path="/var/lib/kubelet/pods/4205b4e6-501d-4aae-9020-3153fdc26b53/volumes"
Sep 08 14:01:02.813192 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:02.813076    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq"]
Sep 08 14:01:02.813758 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:02.813601    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe" containerName="main"
Sep 08 14:01:02.813758 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:02.813619    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe" containerName="main"
Sep 08 14:01:02.813758 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:02.813641    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa" containerName="main"
Sep 08 14:01:02.813758 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:02.813650    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa" containerName="main"
Sep 08 14:01:02.813758 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:02.813675    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4205b4e6-501d-4aae-9020-3153fdc26b53" containerName="main"
Sep 08 14:01:02.813758 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:02.813681    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4205b4e6-501d-4aae-9020-3153fdc26b53" containerName="main"
Sep 08 14:01:02.813758 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:02.813762    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3acda7ac-ea5e-47b8-9fd9-3f83ef7a57aa" containerName="main"
Sep 08 14:01:02.814125 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:02.813770    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4205b4e6-501d-4aae-9020-3153fdc26b53" containerName="main"
Sep 08 14:01:02.814125 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:02.813777    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e06b8ed-7757-4b07-a7e3-0b0f1e561ebe" containerName="main"
Sep 08 14:01:02.817026 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:02.817008    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq"
Sep 08 14:01:02.820879 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:02.820852    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 08 14:01:02.821012 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:02.820858    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-8cxq9\""
Sep 08 14:01:02.821012 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:02.820932    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 08 14:01:02.821012 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:02.820860    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-qbkpr\""
Sep 08 14:01:02.821187 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:02.820871    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 08 14:01:02.831945 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:02.831915    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq"]
Sep 08 14:01:02.929880 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:02.929844    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2q98\" (UniqueName: \"kubernetes.io/projected/db7c2518-de5d-4159-8ef5-9ce22e465efb-kube-api-access-j2q98\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq\" (UID: \"db7c2518-de5d-4159-8ef5-9ce22e465efb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq"
Sep 08 14:01:02.930085 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:02.929987    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db7c2518-de5d-4159-8ef5-9ce22e465efb-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq\" (UID: \"db7c2518-de5d-4159-8ef5-9ce22e465efb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq"
Sep 08 14:01:03.031116 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:03.031075    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db7c2518-de5d-4159-8ef5-9ce22e465efb-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq\" (UID: \"db7c2518-de5d-4159-8ef5-9ce22e465efb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq"
Sep 08 14:01:03.031321 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:03.031138    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2q98\" (UniqueName: \"kubernetes.io/projected/db7c2518-de5d-4159-8ef5-9ce22e465efb-kube-api-access-j2q98\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq\" (UID: \"db7c2518-de5d-4159-8ef5-9ce22e465efb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq"
Sep 08 14:01:03.033876 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:03.033848    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db7c2518-de5d-4159-8ef5-9ce22e465efb-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq\" (UID: \"db7c2518-de5d-4159-8ef5-9ce22e465efb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq"
Sep 08 14:01:03.042904 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:03.042854    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2q98\" (UniqueName: \"kubernetes.io/projected/db7c2518-de5d-4159-8ef5-9ce22e465efb-kube-api-access-j2q98\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq\" (UID: \"db7c2518-de5d-4159-8ef5-9ce22e465efb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq"
Sep 08 14:01:03.128861 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:03.128765    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq"
Sep 08 14:01:03.407585 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:03.407497    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq"]
Sep 08 14:01:03.411313 ip-10-0-130-178 kubenswrapper[2561]: W0908 14:01:03.411264    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb7c2518_de5d_4159_8ef5_9ce22e465efb.slice/crio-21565bba2aaf4b58b42ce7522853c7d61dd7c0630adc7a6ad28447256d472cc4 WatchSource:0}: Error finding container 21565bba2aaf4b58b42ce7522853c7d61dd7c0630adc7a6ad28447256d472cc4: Status 404 returned error can't find the container with id 21565bba2aaf4b58b42ce7522853c7d61dd7c0630adc7a6ad28447256d472cc4
Sep 08 14:01:04.013421 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:04.013367    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq" event={"ID":"db7c2518-de5d-4159-8ef5-9ce22e465efb","Type":"ContainerStarted","Data":"40c3336aa21ce7fc37cf3e0fc804a425867f574d85f6af7ca0720c54022f7ae6"}
Sep 08 14:01:04.013833 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:04.013487    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq"
Sep 08 14:01:04.013833 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:04.013519    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq" event={"ID":"db7c2518-de5d-4159-8ef5-9ce22e465efb","Type":"ContainerStarted","Data":"21565bba2aaf4b58b42ce7522853c7d61dd7c0630adc7a6ad28447256d472cc4"}
Sep 08 14:01:04.032282 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:04.032230    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq" podStartSLOduration=2.032213981 podStartE2EDuration="2.032213981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:01:04.030760622 +0000 UTC m=+2695.491405432" watchObservedRunningTime="2026-09-08 14:01:04.032213981 +0000 UTC m=+2695.492858790"
Sep 08 14:01:09.209562 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:09.209531    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 14:01:09.216844 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:09.216815    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 14:01:27.786826 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.786789    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"]
Sep 08 14:01:27.790565 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.790544    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.794503 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.794478    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-sc5kb\""
Sep 08 14:01:27.794627 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.794479    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 08 14:01:27.794803 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.794783    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 08 14:01:27.794916 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.794899    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 08 14:01:27.803138 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.803109    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"]
Sep 08 14:01:27.856670 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.856631    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-podinfo\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.856670 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.856667    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-token\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.856901 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.856687    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-envoy\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.856901 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.856703    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-data\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.856901 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.856823    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-workload-certs\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.856901 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.856865    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-workload-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.856901 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.856886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv59r\" (UniqueName: \"kubernetes.io/projected/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-kube-api-access-sv59r\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.857100 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.856933    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-credential-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.857100 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.856952    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.958091 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.958042    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-workload-certs\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.958091 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.958095    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-workload-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.958425 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.958121    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv59r\" (UniqueName: \"kubernetes.io/projected/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-kube-api-access-sv59r\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.958425 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.958164    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-credential-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.958425 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.958191    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.958425 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.958240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-podinfo\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.958425 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.958263    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-token\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.958425 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.958325    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-envoy\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.958425 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.958350    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-data\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.958773 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.958568    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-workload-certs\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.958773 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.958591    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-workload-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.958773 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.958657    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-credential-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.958773 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.958741    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-data\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.959104 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.959085    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.960782 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.960763    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-envoy\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.960941 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.960921    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-podinfo\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.966404 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.966380    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-token\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:27.966512 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:27.966489    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv59r\" (UniqueName: \"kubernetes.io/projected/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-kube-api-access-sv59r\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-xclh5\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:28.104240 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:28.104132    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:28.386947 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:28.386919    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"]
Sep 08 14:01:28.389660 ip-10-0-130-178 kubenswrapper[2561]: W0908 14:01:28.389625    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f11b7ab_c4b1_4d93_90cb_c102386aa79e.slice/crio-75903a7c686d1fdf92b8fb809c810cbfc71079a072ffbc5f809d9f7dd83dba11 WatchSource:0}: Error finding container 75903a7c686d1fdf92b8fb809c810cbfc71079a072ffbc5f809d9f7dd83dba11: Status 404 returned error can't find the container with id 75903a7c686d1fdf92b8fb809c810cbfc71079a072ffbc5f809d9f7dd83dba11
Sep 08 14:01:28.391956 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:28.391914    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 14:01:28.392050 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:28.392010    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 14:01:28.392094 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:28.392052    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 14:01:29.146392 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:29.146355    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5" event={"ID":"5f11b7ab-c4b1-4d93-90cb-c102386aa79e","Type":"ContainerStarted","Data":"f97a642a5a55a62691d4b4589d32bc833f0fbaa7ac0f20e303bf1ab88b670b9e"}
Sep 08 14:01:29.146857 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:29.146411    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5" event={"ID":"5f11b7ab-c4b1-4d93-90cb-c102386aa79e","Type":"ContainerStarted","Data":"75903a7c686d1fdf92b8fb809c810cbfc71079a072ffbc5f809d9f7dd83dba11"}
Sep 08 14:01:29.171648 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:29.171594    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5" podStartSLOduration=2.171579165 podStartE2EDuration="2.171579165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:01:29.169697146 +0000 UTC m=+2720.630341957" watchObservedRunningTime="2026-09-08 14:01:29.171579165 +0000 UTC m=+2720.632223971"
Sep 08 14:01:30.105277 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:30.105231    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:30.111019 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:30.110987    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:30.152215 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:30.152175    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:30.153507 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:30.153482    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:01:35.020073 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:35.020040    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq"
Sep 08 14:01:58.318455 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:58.318416    2561 generic.go:356] "Generic (PLEG): container finished" podID="5f11b7ab-c4b1-4d93-90cb-c102386aa79e" containerID="f97a642a5a55a62691d4b4589d32bc833f0fbaa7ac0f20e303bf1ab88b670b9e" exitCode=0
Sep 08 14:01:58.318872 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:58.318505    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5" event={"ID":"5f11b7ab-c4b1-4d93-90cb-c102386aa79e","Type":"ContainerDied","Data":"f97a642a5a55a62691d4b4589d32bc833f0fbaa7ac0f20e303bf1ab88b670b9e"}
Sep 08 14:01:58.318991 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:58.318974    2561 scope.go:117] "RemoveContainer" containerID="f97a642a5a55a62691d4b4589d32bc833f0fbaa7ac0f20e303bf1ab88b670b9e"
Sep 08 14:01:58.320111 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:58.320083    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 14:01:58.320171 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:58.320154    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 14:01:58.320208 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:58.320189    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 14:01:59.326213 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:01:59.326178    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5" event={"ID":"5f11b7ab-c4b1-4d93-90cb-c102386aa79e","Type":"ContainerStarted","Data":"bf08f6b476de8685893e79736fc0798f448ca8f4b46f521a5b93d111f494e931"}
Sep 08 14:02:00.104863 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.104823    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:02:00.106053 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.106018    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5" podUID="5f11b7ab-c4b1-4d93-90cb-c102386aa79e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 08 14:02:00.159314 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.153393    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:02:00.167831 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.166610    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"]
Sep 08 14:02:00.170025 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.169978    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:00.172633 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.172599    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"]
Sep 08 14:02:00.172830 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.172809    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 08 14:02:00.267368 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.267324    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:00.267368 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.267366    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:00.267648 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.267388    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:00.267648 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.267409    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:00.267648 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.267497    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxg9b\" (UniqueName: \"kubernetes.io/projected/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-kube-api-access-mxg9b\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:00.267648 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.267618    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:00.368713 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.368610    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:00.368713 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.368678    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:00.368713 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.368712    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:00.369312 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.368733    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:00.369312 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.368749    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:00.369312 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.368782    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxg9b\" (UniqueName: \"kubernetes.io/projected/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-kube-api-access-mxg9b\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:00.369312 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.369134    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:00.369312 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.369152    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:00.369312 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.369196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:00.371150 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.371132    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:00.371694 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.371667    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:00.382806 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.382778    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxg9b\" (UniqueName: \"kubernetes.io/projected/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-kube-api-access-mxg9b\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:00.483310 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.483268    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:00.767699 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:00.767657    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"]
Sep 08 14:02:00.770884 ip-10-0-130-178 kubenswrapper[2561]: W0908 14:02:00.770846    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6dcc0d_e9f2_4261_8bda_a3d2df25e937.slice/crio-0f4f0d5b21fdc20ffed3921e081c032f43a3f049d1232108ed72fa9a75dc45d8 WatchSource:0}: Error finding container 0f4f0d5b21fdc20ffed3921e081c032f43a3f049d1232108ed72fa9a75dc45d8: Status 404 returned error can't find the container with id 0f4f0d5b21fdc20ffed3921e081c032f43a3f049d1232108ed72fa9a75dc45d8
Sep 08 14:02:01.105333 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:01.105211    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5" podUID="5f11b7ab-c4b1-4d93-90cb-c102386aa79e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 08 14:02:01.339273 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:01.339232    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65" event={"ID":"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937","Type":"ContainerStarted","Data":"1c6f0779189c59c4d90b5e8996a983a7f6b1daa4ba5695b39c52ac47700eb276"}
Sep 08 14:02:01.339475 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:01.339310    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65" event={"ID":"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937","Type":"ContainerStarted","Data":"0f4f0d5b21fdc20ffed3921e081c032f43a3f049d1232108ed72fa9a75dc45d8"}
Sep 08 14:02:01.360805 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:01.360669    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65" podStartSLOduration=1.360645635 podStartE2EDuration="1.360645635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:02:01.358831716 +0000 UTC m=+2752.819476527" watchObservedRunningTime="2026-09-08 14:02:01.360645635 +0000 UTC m=+2752.821290445"
Sep 08 14:02:01.483687 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:01.483653    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:02.105242 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:02.105189    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5" podUID="5f11b7ab-c4b1-4d93-90cb-c102386aa79e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 08 14:02:03.105174 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:03.105128    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5" podUID="5f11b7ab-c4b1-4d93-90cb-c102386aa79e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 08 14:02:04.104727 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:04.104680    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5" podUID="5f11b7ab-c4b1-4d93-90cb-c102386aa79e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 08 14:02:05.109256 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:05.109220    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:02:05.110526 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:05.110504    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:02:10.483798 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:10.483760    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:10.496633 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:10.496601    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:11.411859 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:11.411818    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:33.877504 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:33.877415    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"]
Sep 08 14:02:33.887002 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:02:33.886960    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 14:02:33.887002 ip-10-0-130-178 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 14:02:33.887002 ip-10-0-130-178 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 08 14:02:33.887002 ip-10-0-130-178 kubenswrapper[2561]:  > containerID="1c6f0779189c59c4d90b5e8996a983a7f6b1daa4ba5695b39c52ac47700eb276" cmd=["/bin/sleep","15"]
Sep 08 14:02:33.887395 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:33.887018    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65" message="" err=<
Sep 08 14:02:33.887395 ip-10-0-130-178 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 14:02:33.887395 ip-10-0-130-178 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 08 14:02:33.887395 ip-10-0-130-178 kubenswrapper[2561]:  >
Sep 08 14:02:33.887395 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:02:33.887037    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 14:02:33.887395 ip-10-0-130-178 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 14:02:33.887395 ip-10-0-130-178 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 08 14:02:33.887395 ip-10-0-130-178 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65" podUID="9c6dcc0d-e9f2-4261-8bda-a3d2df25e937" containerName="main" containerID="cri-o://1c6f0779189c59c4d90b5e8996a983a7f6b1daa4ba5695b39c52ac47700eb276"
Sep 08 14:02:33.887395 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:33.887086    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65" podUID="9c6dcc0d-e9f2-4261-8bda-a3d2df25e937" containerName="main" containerID="cri-o://1c6f0779189c59c4d90b5e8996a983a7f6b1daa4ba5695b39c52ac47700eb276" gracePeriod=60
Sep 08 14:02:34.244953 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.244925    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:34.298471 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.298434    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-tmp-dir\") pod \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") "
Sep 08 14:02:34.298471 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.298477    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-tls-certs\") pod \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") "
Sep 08 14:02:34.298745 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.298555    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-model-cache\") pod \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") "
Sep 08 14:02:34.298745 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.298575    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxg9b\" (UniqueName: \"kubernetes.io/projected/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-kube-api-access-mxg9b\") pod \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") "
Sep 08 14:02:34.298745 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.298603    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-dshm\") pod \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") "
Sep 08 14:02:34.298745 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.298654    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-home\") pod \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\" (UID: \"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937\") "
Sep 08 14:02:34.298944 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.298775    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c6dcc0d-e9f2-4261-8bda-a3d2df25e937" (UID: "9c6dcc0d-e9f2-4261-8bda-a3d2df25e937"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:02:34.298944 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.298844    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c6dcc0d-e9f2-4261-8bda-a3d2df25e937" (UID: "9c6dcc0d-e9f2-4261-8bda-a3d2df25e937"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:02:34.299054 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.298978    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-tmp-dir\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:02:34.299054 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.298999    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-model-cache\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:02:34.299227 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.299157    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-home" (OuterVolumeSpecName: "home") pod "9c6dcc0d-e9f2-4261-8bda-a3d2df25e937" (UID: "9c6dcc0d-e9f2-4261-8bda-a3d2df25e937"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:02:34.301203 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.301174    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-dshm" (OuterVolumeSpecName: "dshm") pod "9c6dcc0d-e9f2-4261-8bda-a3d2df25e937" (UID: "9c6dcc0d-e9f2-4261-8bda-a3d2df25e937"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:02:34.301371 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.301353    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-kube-api-access-mxg9b" (OuterVolumeSpecName: "kube-api-access-mxg9b") pod "9c6dcc0d-e9f2-4261-8bda-a3d2df25e937" (UID: "9c6dcc0d-e9f2-4261-8bda-a3d2df25e937"). InnerVolumeSpecName "kube-api-access-mxg9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:02:34.301427 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.301353    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c6dcc0d-e9f2-4261-8bda-a3d2df25e937" (UID: "9c6dcc0d-e9f2-4261-8bda-a3d2df25e937"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:02:34.400343 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.400262    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-home\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:02:34.400343 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.400336    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:02:34.400343 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.400349    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxg9b\" (UniqueName: \"kubernetes.io/projected/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-kube-api-access-mxg9b\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:02:34.400591 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.400357    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937-dshm\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:02:34.537149 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.537114    2561 generic.go:356] "Generic (PLEG): container finished" podID="9c6dcc0d-e9f2-4261-8bda-a3d2df25e937" containerID="1c6f0779189c59c4d90b5e8996a983a7f6b1daa4ba5695b39c52ac47700eb276" exitCode=0
Sep 08 14:02:34.537361 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.537161    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"
Sep 08 14:02:34.537361 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.537194    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65" event={"ID":"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937","Type":"ContainerDied","Data":"1c6f0779189c59c4d90b5e8996a983a7f6b1daa4ba5695b39c52ac47700eb276"}
Sep 08 14:02:34.537361 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.537255    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65" event={"ID":"9c6dcc0d-e9f2-4261-8bda-a3d2df25e937","Type":"ContainerDied","Data":"0f4f0d5b21fdc20ffed3921e081c032f43a3f049d1232108ed72fa9a75dc45d8"}
Sep 08 14:02:34.537361 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.537315    2561 scope.go:117] "RemoveContainer" containerID="1c6f0779189c59c4d90b5e8996a983a7f6b1daa4ba5695b39c52ac47700eb276"
Sep 08 14:02:34.548476 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.548449    2561 scope.go:117] "RemoveContainer" containerID="1c6f0779189c59c4d90b5e8996a983a7f6b1daa4ba5695b39c52ac47700eb276"
Sep 08 14:02:34.548844 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:02:34.548818    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c6f0779189c59c4d90b5e8996a983a7f6b1daa4ba5695b39c52ac47700eb276\": container with ID starting with 1c6f0779189c59c4d90b5e8996a983a7f6b1daa4ba5695b39c52ac47700eb276 not found: ID does not exist" containerID="1c6f0779189c59c4d90b5e8996a983a7f6b1daa4ba5695b39c52ac47700eb276"
Sep 08 14:02:34.548915 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.548851    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c6f0779189c59c4d90b5e8996a983a7f6b1daa4ba5695b39c52ac47700eb276"} err="failed to get container status \"1c6f0779189c59c4d90b5e8996a983a7f6b1daa4ba5695b39c52ac47700eb276\": rpc error: code = NotFound desc = could not find container \"1c6f0779189c59c4d90b5e8996a983a7f6b1daa4ba5695b39c52ac47700eb276\": container with ID starting with 1c6f0779189c59c4d90b5e8996a983a7f6b1daa4ba5695b39c52ac47700eb276 not found: ID does not exist"
Sep 08 14:02:34.564812 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.564761    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"]
Sep 08 14:02:34.572851 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:34.572815    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5hp65"]
Sep 08 14:02:35.007639 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:35.007593    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c6dcc0d-e9f2-4261-8bda-a3d2df25e937" path="/var/lib/kubelet/pods/9c6dcc0d-e9f2-4261-8bda-a3d2df25e937/volumes"
Sep 08 14:02:47.296522 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:47.296481    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"]
Sep 08 14:02:47.297043 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:47.296702    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5" podUID="5f11b7ab-c4b1-4d93-90cb-c102386aa79e" containerName="istio-proxy" containerID="cri-o://bf08f6b476de8685893e79736fc0798f448ca8f4b46f521a5b93d111f494e931" gracePeriod=30
Sep 08 14:02:50.112010 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:50.111954    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5" podUID="5f11b7ab-c4b1-4d93-90cb-c102386aa79e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 14:02:50.572549 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:50.572514    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7"]
Sep 08 14:02:50.572932 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:50.572915    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c6dcc0d-e9f2-4261-8bda-a3d2df25e937" containerName="main"
Sep 08 14:02:50.572983 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:50.572934    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c6dcc0d-e9f2-4261-8bda-a3d2df25e937" containerName="main"
Sep 08 14:02:50.573036 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:50.573002    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c6dcc0d-e9f2-4261-8bda-a3d2df25e937" containerName="main"
Sep 08 14:02:50.576174 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:50.576155    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7"
Sep 08 14:02:50.578521 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:50.578500    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 08 14:02:50.578644 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:50.578626    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 08 14:02:50.579461 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:50.579437    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-qzxkd\""
Sep 08 14:02:50.579584 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:50.579434    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 08 14:02:50.579584 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:50.579560    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-krkns\""
Sep 08 14:02:50.586608 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:50.586584    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7"]
Sep 08 14:02:50.752667 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:50.752624    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae57c095-4306-45d2-a308-bdeb89ba3d06-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7\" (UID: \"ae57c095-4306-45d2-a308-bdeb89ba3d06\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7"
Sep 08 14:02:50.752853 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:50.752674    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt25f\" (UniqueName: \"kubernetes.io/projected/ae57c095-4306-45d2-a308-bdeb89ba3d06-kube-api-access-qt25f\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7\" (UID: \"ae57c095-4306-45d2-a308-bdeb89ba3d06\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7"
Sep 08 14:02:50.853948 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:50.853859    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt25f\" (UniqueName: \"kubernetes.io/projected/ae57c095-4306-45d2-a308-bdeb89ba3d06-kube-api-access-qt25f\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7\" (UID: \"ae57c095-4306-45d2-a308-bdeb89ba3d06\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7"
Sep 08 14:02:50.854117 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:50.853967    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae57c095-4306-45d2-a308-bdeb89ba3d06-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7\" (UID: \"ae57c095-4306-45d2-a308-bdeb89ba3d06\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7"
Sep 08 14:02:50.856468 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:50.856438    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae57c095-4306-45d2-a308-bdeb89ba3d06-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7\" (UID: \"ae57c095-4306-45d2-a308-bdeb89ba3d06\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7"
Sep 08 14:02:50.862362 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:50.862338    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt25f\" (UniqueName: \"kubernetes.io/projected/ae57c095-4306-45d2-a308-bdeb89ba3d06-kube-api-access-qt25f\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7\" (UID: \"ae57c095-4306-45d2-a308-bdeb89ba3d06\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7"
Sep 08 14:02:50.887397 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:50.887365    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7"
Sep 08 14:02:51.175306 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:51.175261    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7"]
Sep 08 14:02:51.177477 ip-10-0-130-178 kubenswrapper[2561]: W0908 14:02:51.177449    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae57c095_4306_45d2_a308_bdeb89ba3d06.slice/crio-24fe2b6aacd7a02cb9b973846875718da0524b9dc4cbe31566a5dfb33f33c643 WatchSource:0}: Error finding container 24fe2b6aacd7a02cb9b973846875718da0524b9dc4cbe31566a5dfb33f33c643: Status 404 returned error can't find the container with id 24fe2b6aacd7a02cb9b973846875718da0524b9dc4cbe31566a5dfb33f33c643
Sep 08 14:02:51.632083 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:51.632041    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7" event={"ID":"ae57c095-4306-45d2-a308-bdeb89ba3d06","Type":"ContainerStarted","Data":"673986bef243816bcedc0d28760af016959fac06a15922f37665721d4cfbc198"}
Sep 08 14:02:51.632319 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:51.632100    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7" event={"ID":"ae57c095-4306-45d2-a308-bdeb89ba3d06","Type":"ContainerStarted","Data":"24fe2b6aacd7a02cb9b973846875718da0524b9dc4cbe31566a5dfb33f33c643"}
Sep 08 14:02:51.632319 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:51.632180    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7"
Sep 08 14:02:51.657024 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:51.656962    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7" podStartSLOduration=1.656943261 podStartE2EDuration="1.656943261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:02:51.654137632 +0000 UTC m=+2803.114782441" watchObservedRunningTime="2026-09-08 14:02:51.656943261 +0000 UTC m=+2803.117588068"
Sep 08 14:02:52.561086 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.561015    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:02:52.640381 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.640344    2561 generic.go:356] "Generic (PLEG): container finished" podID="5f11b7ab-c4b1-4d93-90cb-c102386aa79e" containerID="bf08f6b476de8685893e79736fc0798f448ca8f4b46f521a5b93d111f494e931" exitCode=0
Sep 08 14:02:52.640582 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.640396    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5" event={"ID":"5f11b7ab-c4b1-4d93-90cb-c102386aa79e","Type":"ContainerDied","Data":"bf08f6b476de8685893e79736fc0798f448ca8f4b46f521a5b93d111f494e931"}
Sep 08 14:02:52.640582 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.640451    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"
Sep 08 14:02:52.640582 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.640464    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5" event={"ID":"5f11b7ab-c4b1-4d93-90cb-c102386aa79e","Type":"ContainerDied","Data":"75903a7c686d1fdf92b8fb809c810cbfc71079a072ffbc5f809d9f7dd83dba11"}
Sep 08 14:02:52.640582 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.640496    2561 scope.go:117] "RemoveContainer" containerID="bf08f6b476de8685893e79736fc0798f448ca8f4b46f521a5b93d111f494e931"
Sep 08 14:02:52.653331 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.653302    2561 scope.go:117] "RemoveContainer" containerID="f97a642a5a55a62691d4b4589d32bc833f0fbaa7ac0f20e303bf1ab88b670b9e"
Sep 08 14:02:52.662477 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.662453    2561 scope.go:117] "RemoveContainer" containerID="bf08f6b476de8685893e79736fc0798f448ca8f4b46f521a5b93d111f494e931"
Sep 08 14:02:52.662800 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:02:52.662774    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf08f6b476de8685893e79736fc0798f448ca8f4b46f521a5b93d111f494e931\": container with ID starting with bf08f6b476de8685893e79736fc0798f448ca8f4b46f521a5b93d111f494e931 not found: ID does not exist" containerID="bf08f6b476de8685893e79736fc0798f448ca8f4b46f521a5b93d111f494e931"
Sep 08 14:02:52.662861 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.662814    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf08f6b476de8685893e79736fc0798f448ca8f4b46f521a5b93d111f494e931"} err="failed to get container status \"bf08f6b476de8685893e79736fc0798f448ca8f4b46f521a5b93d111f494e931\": rpc error: code = NotFound desc = could not find container \"bf08f6b476de8685893e79736fc0798f448ca8f4b46f521a5b93d111f494e931\": container with ID starting with bf08f6b476de8685893e79736fc0798f448ca8f4b46f521a5b93d111f494e931 not found: ID does not exist"
Sep 08 14:02:52.662861 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.662840    2561 scope.go:117] "RemoveContainer" containerID="f97a642a5a55a62691d4b4589d32bc833f0fbaa7ac0f20e303bf1ab88b670b9e"
Sep 08 14:02:52.663347 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:02:52.663316    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f97a642a5a55a62691d4b4589d32bc833f0fbaa7ac0f20e303bf1ab88b670b9e\": container with ID starting with f97a642a5a55a62691d4b4589d32bc833f0fbaa7ac0f20e303bf1ab88b670b9e not found: ID does not exist" containerID="f97a642a5a55a62691d4b4589d32bc833f0fbaa7ac0f20e303bf1ab88b670b9e"
Sep 08 14:02:52.663447 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.663347    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f97a642a5a55a62691d4b4589d32bc833f0fbaa7ac0f20e303bf1ab88b670b9e"} err="failed to get container status \"f97a642a5a55a62691d4b4589d32bc833f0fbaa7ac0f20e303bf1ab88b670b9e\": rpc error: code = NotFound desc = could not find container \"f97a642a5a55a62691d4b4589d32bc833f0fbaa7ac0f20e303bf1ab88b670b9e\": container with ID starting with f97a642a5a55a62691d4b4589d32bc833f0fbaa7ac0f20e303bf1ab88b670b9e not found: ID does not exist"
Sep 08 14:02:52.670581 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.670556    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-credential-socket\") pod \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") "
Sep 08 14:02:52.670700 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.670602    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-workload-socket\") pod \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") "
Sep 08 14:02:52.670700 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.670638    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-token\") pod \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") "
Sep 08 14:02:52.670700 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.670682    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-data\") pod \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") "
Sep 08 14:02:52.670873 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.670701    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istiod-ca-cert\") pod \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") "
Sep 08 14:02:52.670873 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.670725    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv59r\" (UniqueName: \"kubernetes.io/projected/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-kube-api-access-sv59r\") pod \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") "
Sep 08 14:02:52.670873 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.670762    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-podinfo\") pod \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") "
Sep 08 14:02:52.670873 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.670797    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5f11b7ab-c4b1-4d93-90cb-c102386aa79e" (UID: "5f11b7ab-c4b1-4d93-90cb-c102386aa79e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:02:52.671077 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.670916    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-envoy\") pod \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") "
Sep 08 14:02:52.671077 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.670932    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5f11b7ab-c4b1-4d93-90cb-c102386aa79e" (UID: "5f11b7ab-c4b1-4d93-90cb-c102386aa79e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:02:52.671077 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.670948    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-workload-certs\") pod \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\" (UID: \"5f11b7ab-c4b1-4d93-90cb-c102386aa79e\") "
Sep 08 14:02:52.671077 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.671033    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-data" (OuterVolumeSpecName: "istio-data") pod "5f11b7ab-c4b1-4d93-90cb-c102386aa79e" (UID: "5f11b7ab-c4b1-4d93-90cb-c102386aa79e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:02:52.671265 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.671135    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5f11b7ab-c4b1-4d93-90cb-c102386aa79e" (UID: "5f11b7ab-c4b1-4d93-90cb-c102386aa79e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:02:52.671265 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.671190    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5f11b7ab-c4b1-4d93-90cb-c102386aa79e" (UID: "5f11b7ab-c4b1-4d93-90cb-c102386aa79e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:02:52.671401 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.671372    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-data\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:02:52.671401 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.671393    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istiod-ca-cert\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:02:52.671504 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.671409    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-workload-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:02:52.671504 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.671423    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-credential-socket\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:02:52.671504 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.671438    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-workload-socket\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:02:52.673184 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.673157    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5f11b7ab-c4b1-4d93-90cb-c102386aa79e" (UID: "5f11b7ab-c4b1-4d93-90cb-c102386aa79e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 14:02:52.673276 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.673209    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5f11b7ab-c4b1-4d93-90cb-c102386aa79e" (UID: "5f11b7ab-c4b1-4d93-90cb-c102386aa79e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:02:52.673476 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.673453    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-kube-api-access-sv59r" (OuterVolumeSpecName: "kube-api-access-sv59r") pod "5f11b7ab-c4b1-4d93-90cb-c102386aa79e" (UID: "5f11b7ab-c4b1-4d93-90cb-c102386aa79e"). InnerVolumeSpecName "kube-api-access-sv59r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:02:52.673567 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.673482    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-token" (OuterVolumeSpecName: "istio-token") pod "5f11b7ab-c4b1-4d93-90cb-c102386aa79e" (UID: "5f11b7ab-c4b1-4d93-90cb-c102386aa79e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:02:52.771926 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.771886    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-envoy\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:02:52.771926 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.771922    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-token\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:02:52.771926 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.771938    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv59r\" (UniqueName: \"kubernetes.io/projected/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-kube-api-access-sv59r\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:02:52.772167 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.771948    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f11b7ab-c4b1-4d93-90cb-c102386aa79e-istio-podinfo\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:02:52.972093 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.972048    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"]
Sep 08 14:02:52.977801 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:52.977768    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-xclh5"]
Sep 08 14:02:53.007734 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:02:53.007699    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f11b7ab-c4b1-4d93-90cb-c102386aa79e" path="/var/lib/kubelet/pods/5f11b7ab-c4b1-4d93-90cb-c102386aa79e/volumes"
Sep 08 14:03:22.643083 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:03:22.643051    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7"
Sep 08 14:03:23.986250 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:03:23.986202    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq"]
Sep 08 14:03:25.020204 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:03:25.020161    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq" podUID="db7c2518-de5d-4159-8ef5-9ce22e465efb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:03:35.020516 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:03:35.020471    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq" podUID="db7c2518-de5d-4159-8ef5-9ce22e465efb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:03:39.008022 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:03:39.007966    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq" podUID="db7c2518-de5d-4159-8ef5-9ce22e465efb" containerName="main" containerID="cri-o://40c3336aa21ce7fc37cf3e0fc804a425867f574d85f6af7ca0720c54022f7ae6" gracePeriod=45
Sep 08 14:03:45.020542 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:03:45.020501    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq" podUID="db7c2518-de5d-4159-8ef5-9ce22e465efb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:03:45.020978 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:03:45.020624    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq"
Sep 08 14:03:55.020217 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:03:55.020177    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq" podUID="db7c2518-de5d-4159-8ef5-9ce22e465efb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:04:05.020752 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:05.020670    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq" podUID="db7c2518-de5d-4159-8ef5-9ce22e465efb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:04:09.261889 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:09.261862    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq"
Sep 08 14:04:09.352258 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:09.352169    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2q98\" (UniqueName: \"kubernetes.io/projected/db7c2518-de5d-4159-8ef5-9ce22e465efb-kube-api-access-j2q98\") pod \"db7c2518-de5d-4159-8ef5-9ce22e465efb\" (UID: \"db7c2518-de5d-4159-8ef5-9ce22e465efb\") "
Sep 08 14:04:09.352258 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:09.352219    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db7c2518-de5d-4159-8ef5-9ce22e465efb-tls-certs\") pod \"db7c2518-de5d-4159-8ef5-9ce22e465efb\" (UID: \"db7c2518-de5d-4159-8ef5-9ce22e465efb\") "
Sep 08 14:04:09.354372 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:09.354342    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db7c2518-de5d-4159-8ef5-9ce22e465efb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db7c2518-de5d-4159-8ef5-9ce22e465efb" (UID: "db7c2518-de5d-4159-8ef5-9ce22e465efb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:04:09.354509 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:09.354450    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db7c2518-de5d-4159-8ef5-9ce22e465efb-kube-api-access-j2q98" (OuterVolumeSpecName: "kube-api-access-j2q98") pod "db7c2518-de5d-4159-8ef5-9ce22e465efb" (UID: "db7c2518-de5d-4159-8ef5-9ce22e465efb"). InnerVolumeSpecName "kube-api-access-j2q98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:04:09.453883 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:09.453838    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2q98\" (UniqueName: \"kubernetes.io/projected/db7c2518-de5d-4159-8ef5-9ce22e465efb-kube-api-access-j2q98\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:04:09.453883 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:09.453875    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db7c2518-de5d-4159-8ef5-9ce22e465efb-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:04:10.049748 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:10.049716    2561 generic.go:356] "Generic (PLEG): container finished" podID="db7c2518-de5d-4159-8ef5-9ce22e465efb" containerID="40c3336aa21ce7fc37cf3e0fc804a425867f574d85f6af7ca0720c54022f7ae6" exitCode=0
Sep 08 14:04:10.049927 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:10.049760    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq"
Sep 08 14:04:10.049927 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:10.049781    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq" event={"ID":"db7c2518-de5d-4159-8ef5-9ce22e465efb","Type":"ContainerDied","Data":"40c3336aa21ce7fc37cf3e0fc804a425867f574d85f6af7ca0720c54022f7ae6"}
Sep 08 14:04:10.049927 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:10.049824    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq" event={"ID":"db7c2518-de5d-4159-8ef5-9ce22e465efb","Type":"ContainerDied","Data":"21565bba2aaf4b58b42ce7522853c7d61dd7c0630adc7a6ad28447256d472cc4"}
Sep 08 14:04:10.049927 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:10.049844    2561 scope.go:117] "RemoveContainer" containerID="40c3336aa21ce7fc37cf3e0fc804a425867f574d85f6af7ca0720c54022f7ae6"
Sep 08 14:04:10.058980 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:10.058961    2561 scope.go:117] "RemoveContainer" containerID="40c3336aa21ce7fc37cf3e0fc804a425867f574d85f6af7ca0720c54022f7ae6"
Sep 08 14:04:10.059318 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:04:10.059278    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c3336aa21ce7fc37cf3e0fc804a425867f574d85f6af7ca0720c54022f7ae6\": container with ID starting with 40c3336aa21ce7fc37cf3e0fc804a425867f574d85f6af7ca0720c54022f7ae6 not found: ID does not exist" containerID="40c3336aa21ce7fc37cf3e0fc804a425867f574d85f6af7ca0720c54022f7ae6"
Sep 08 14:04:10.059383 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:10.059330    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c3336aa21ce7fc37cf3e0fc804a425867f574d85f6af7ca0720c54022f7ae6"} err="failed to get container status \"40c3336aa21ce7fc37cf3e0fc804a425867f574d85f6af7ca0720c54022f7ae6\": rpc error: code = NotFound desc = could not find container \"40c3336aa21ce7fc37cf3e0fc804a425867f574d85f6af7ca0720c54022f7ae6\": container with ID starting with 40c3336aa21ce7fc37cf3e0fc804a425867f574d85f6af7ca0720c54022f7ae6 not found: ID does not exist"
Sep 08 14:04:10.074115 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:10.074076    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq"]
Sep 08 14:04:10.079620 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:10.079595    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheg9jfq"]
Sep 08 14:04:11.007208 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:11.007171    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db7c2518-de5d-4159-8ef5-9ce22e465efb" path="/var/lib/kubelet/pods/db7c2518-de5d-4159-8ef5-9ce22e465efb/volumes"
Sep 08 14:04:24.003779 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.003739    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h"]
Sep 08 14:04:24.004265 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.004147    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f11b7ab-c4b1-4d93-90cb-c102386aa79e" containerName="istio-proxy"
Sep 08 14:04:24.004265 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.004164    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f11b7ab-c4b1-4d93-90cb-c102386aa79e" containerName="istio-proxy"
Sep 08 14:04:24.004265 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.004176    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db7c2518-de5d-4159-8ef5-9ce22e465efb" containerName="main"
Sep 08 14:04:24.004265 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.004182    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="db7c2518-de5d-4159-8ef5-9ce22e465efb" containerName="main"
Sep 08 14:04:24.004265 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.004193    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f11b7ab-c4b1-4d93-90cb-c102386aa79e" containerName="istio-proxy"
Sep 08 14:04:24.004265 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.004198    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f11b7ab-c4b1-4d93-90cb-c102386aa79e" containerName="istio-proxy"
Sep 08 14:04:24.004265 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.004259    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f11b7ab-c4b1-4d93-90cb-c102386aa79e" containerName="istio-proxy"
Sep 08 14:04:24.004265 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.004266    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f11b7ab-c4b1-4d93-90cb-c102386aa79e" containerName="istio-proxy"
Sep 08 14:04:24.004696 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.004274    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="db7c2518-de5d-4159-8ef5-9ce22e465efb" containerName="main"
Sep 08 14:04:24.007267 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.007245    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h"
Sep 08 14:04:24.010468 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.010438    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 08 14:04:24.010602 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.010489    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 08 14:04:24.010602 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.010447    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-8mjm6\""
Sep 08 14:04:24.010602 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.010532    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 08 14:04:24.010793 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.010780    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-9825j\""
Sep 08 14:04:24.019169 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.019143    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h"]
Sep 08 14:04:24.084495 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.084457    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c6e450-4c8b-47d1-adcd-68d2460c086d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h\" (UID: \"94c6e450-4c8b-47d1-adcd-68d2460c086d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h"
Sep 08 14:04:24.084684 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.084541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjfvs\" (UniqueName: \"kubernetes.io/projected/94c6e450-4c8b-47d1-adcd-68d2460c086d-kube-api-access-jjfvs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h\" (UID: \"94c6e450-4c8b-47d1-adcd-68d2460c086d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h"
Sep 08 14:04:24.185712 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.185671    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjfvs\" (UniqueName: \"kubernetes.io/projected/94c6e450-4c8b-47d1-adcd-68d2460c086d-kube-api-access-jjfvs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h\" (UID: \"94c6e450-4c8b-47d1-adcd-68d2460c086d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h"
Sep 08 14:04:24.185879 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.185735    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c6e450-4c8b-47d1-adcd-68d2460c086d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h\" (UID: \"94c6e450-4c8b-47d1-adcd-68d2460c086d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h"
Sep 08 14:04:24.188324 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.188300    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c6e450-4c8b-47d1-adcd-68d2460c086d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h\" (UID: \"94c6e450-4c8b-47d1-adcd-68d2460c086d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h"
Sep 08 14:04:24.195264 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.195240    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjfvs\" (UniqueName: \"kubernetes.io/projected/94c6e450-4c8b-47d1-adcd-68d2460c086d-kube-api-access-jjfvs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h\" (UID: \"94c6e450-4c8b-47d1-adcd-68d2460c086d\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h"
Sep 08 14:04:24.318936 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.318818    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h"
Sep 08 14:04:24.595704 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:24.595678    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h"]
Sep 08 14:04:24.597827 ip-10-0-130-178 kubenswrapper[2561]: W0908 14:04:24.597801    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c6e450_4c8b_47d1_adcd_68d2460c086d.slice/crio-0863da69241be6dac6dd95d654ccd969973908c4723cbe8de752604d9b6bd1b9 WatchSource:0}: Error finding container 0863da69241be6dac6dd95d654ccd969973908c4723cbe8de752604d9b6bd1b9: Status 404 returned error can't find the container with id 0863da69241be6dac6dd95d654ccd969973908c4723cbe8de752604d9b6bd1b9
Sep 08 14:04:25.131083 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:25.131035    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h" event={"ID":"94c6e450-4c8b-47d1-adcd-68d2460c086d","Type":"ContainerStarted","Data":"24a18e7e64f5396c9be9220f26fc71809bbe67800e0ff4acb616fdeccd2a9083"}
Sep 08 14:04:25.131632 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:25.131097    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h" event={"ID":"94c6e450-4c8b-47d1-adcd-68d2460c086d","Type":"ContainerStarted","Data":"0863da69241be6dac6dd95d654ccd969973908c4723cbe8de752604d9b6bd1b9"}
Sep 08 14:04:25.131632 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:25.131150    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h"
Sep 08 14:04:25.149959 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:25.149889    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h" podStartSLOduration=2.149866984 podStartE2EDuration="2.149866984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:04:25.149353285 +0000 UTC m=+2896.609998099" watchObservedRunningTime="2026-09-08 14:04:25.149866984 +0000 UTC m=+2896.610511793"
Sep 08 14:04:55.624926 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:55.624888    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7"]
Sep 08 14:04:56.141061 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:04:56.141029    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h"
Sep 08 14:05:02.642655 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:02.642615    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7" podUID="ae57c095-4306-45d2-a308-bdeb89ba3d06" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:05:10.645911 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:10.645865    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7" podUID="ae57c095-4306-45d2-a308-bdeb89ba3d06" containerName="main" containerID="cri-o://673986bef243816bcedc0d28760af016959fac06a15922f37665721d4cfbc198" gracePeriod=45
Sep 08 14:05:12.643128 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:12.643090    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7" podUID="ae57c095-4306-45d2-a308-bdeb89ba3d06" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:05:22.642763 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:22.642722    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7" podUID="ae57c095-4306-45d2-a308-bdeb89ba3d06" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:05:22.643179 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:22.642830    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7"
Sep 08 14:05:32.643062 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:32.642967    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7" podUID="ae57c095-4306-45d2-a308-bdeb89ba3d06" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:05:40.901353 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:40.901320    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7"
Sep 08 14:05:41.069019 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:41.068989    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae57c095-4306-45d2-a308-bdeb89ba3d06-tls-certs\") pod \"ae57c095-4306-45d2-a308-bdeb89ba3d06\" (UID: \"ae57c095-4306-45d2-a308-bdeb89ba3d06\") "
Sep 08 14:05:41.069237 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:41.069031    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qt25f\" (UniqueName: \"kubernetes.io/projected/ae57c095-4306-45d2-a308-bdeb89ba3d06-kube-api-access-qt25f\") pod \"ae57c095-4306-45d2-a308-bdeb89ba3d06\" (UID: \"ae57c095-4306-45d2-a308-bdeb89ba3d06\") "
Sep 08 14:05:41.071372 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:41.071335    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae57c095-4306-45d2-a308-bdeb89ba3d06-kube-api-access-qt25f" (OuterVolumeSpecName: "kube-api-access-qt25f") pod "ae57c095-4306-45d2-a308-bdeb89ba3d06" (UID: "ae57c095-4306-45d2-a308-bdeb89ba3d06"). InnerVolumeSpecName "kube-api-access-qt25f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:05:41.071475 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:41.071377    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae57c095-4306-45d2-a308-bdeb89ba3d06-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae57c095-4306-45d2-a308-bdeb89ba3d06" (UID: "ae57c095-4306-45d2-a308-bdeb89ba3d06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:05:41.169903 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:41.169864    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae57c095-4306-45d2-a308-bdeb89ba3d06-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:05:41.169903 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:41.169898    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt25f\" (UniqueName: \"kubernetes.io/projected/ae57c095-4306-45d2-a308-bdeb89ba3d06-kube-api-access-qt25f\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:05:41.553871 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:41.553833    2561 generic.go:356] "Generic (PLEG): container finished" podID="ae57c095-4306-45d2-a308-bdeb89ba3d06" containerID="673986bef243816bcedc0d28760af016959fac06a15922f37665721d4cfbc198" exitCode=0
Sep 08 14:05:41.554052 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:41.553881    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7"
Sep 08 14:05:41.554052 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:41.553907    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7" event={"ID":"ae57c095-4306-45d2-a308-bdeb89ba3d06","Type":"ContainerDied","Data":"673986bef243816bcedc0d28760af016959fac06a15922f37665721d4cfbc198"}
Sep 08 14:05:41.554052 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:41.553971    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7" event={"ID":"ae57c095-4306-45d2-a308-bdeb89ba3d06","Type":"ContainerDied","Data":"24fe2b6aacd7a02cb9b973846875718da0524b9dc4cbe31566a5dfb33f33c643"}
Sep 08 14:05:41.554052 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:41.553996    2561 scope.go:117] "RemoveContainer" containerID="673986bef243816bcedc0d28760af016959fac06a15922f37665721d4cfbc198"
Sep 08 14:05:41.563374 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:41.563349    2561 scope.go:117] "RemoveContainer" containerID="673986bef243816bcedc0d28760af016959fac06a15922f37665721d4cfbc198"
Sep 08 14:05:41.563687 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:05:41.563667    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"673986bef243816bcedc0d28760af016959fac06a15922f37665721d4cfbc198\": container with ID starting with 673986bef243816bcedc0d28760af016959fac06a15922f37665721d4cfbc198 not found: ID does not exist" containerID="673986bef243816bcedc0d28760af016959fac06a15922f37665721d4cfbc198"
Sep 08 14:05:41.563760 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:41.563701    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"673986bef243816bcedc0d28760af016959fac06a15922f37665721d4cfbc198"} err="failed to get container status \"673986bef243816bcedc0d28760af016959fac06a15922f37665721d4cfbc198\": rpc error: code = NotFound desc = could not find container \"673986bef243816bcedc0d28760af016959fac06a15922f37665721d4cfbc198\": container with ID starting with 673986bef243816bcedc0d28760af016959fac06a15922f37665721d4cfbc198 not found: ID does not exist"
Sep 08 14:05:41.576475 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:41.576449    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7"]
Sep 08 14:05:41.581261 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:41.581237    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexqkz7"]
Sep 08 14:05:43.006317 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:05:43.006270    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae57c095-4306-45d2-a308-bdeb89ba3d06" path="/var/lib/kubelet/pods/ae57c095-4306-45d2-a308-bdeb89ba3d06/volumes"
Sep 08 14:06:04.109928 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:04.109891    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7"]
Sep 08 14:06:04.110527 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:04.110298    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae57c095-4306-45d2-a308-bdeb89ba3d06" containerName="main"
Sep 08 14:06:04.110527 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:04.110317    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae57c095-4306-45d2-a308-bdeb89ba3d06" containerName="main"
Sep 08 14:06:04.110527 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:04.110462    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae57c095-4306-45d2-a308-bdeb89ba3d06" containerName="main"
Sep 08 14:06:04.113702 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:04.113683    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7"
Sep 08 14:06:04.117011 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:04.116984    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 08 14:06:04.117129 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:04.117025    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 08 14:06:04.117129 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:04.117045    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-kjh6z\""
Sep 08 14:06:04.117129 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:04.117061    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 08 14:06:04.117262 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:04.117181    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-r45vf\""
Sep 08 14:06:04.122626 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:04.122606    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7"]
Sep 08 14:06:04.169768 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:04.169730    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdnd7\" (UniqueName: \"kubernetes.io/projected/fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70-kube-api-access-kdnd7\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7\" (UID: \"fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7"
Sep 08 14:06:04.169941 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:04.169775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7\" (UID: \"fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7"
Sep 08 14:06:04.270909 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:04.270867    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdnd7\" (UniqueName: \"kubernetes.io/projected/fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70-kube-api-access-kdnd7\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7\" (UID: \"fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7"
Sep 08 14:06:04.271103 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:04.270930    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7\" (UID: \"fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7"
Sep 08 14:06:04.273926 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:04.273894    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7\" (UID: \"fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7"
Sep 08 14:06:04.281752 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:04.281690    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdnd7\" (UniqueName: \"kubernetes.io/projected/fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70-kube-api-access-kdnd7\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7\" (UID: \"fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7"
Sep 08 14:06:04.425409 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:04.425303    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7"
Sep 08 14:06:04.701422 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:04.701395    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7"]
Sep 08 14:06:04.703627 ip-10-0-130-178 kubenswrapper[2561]: W0908 14:06:04.703598    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd07d7f_4d7e_46f7_8f85_e473cfaa9f70.slice/crio-868177c7a172b7c3bed669cf8f3b6fc71359e4fb59b3f8c1d6bb6aa5a75fd1cb WatchSource:0}: Error finding container 868177c7a172b7c3bed669cf8f3b6fc71359e4fb59b3f8c1d6bb6aa5a75fd1cb: Status 404 returned error can't find the container with id 868177c7a172b7c3bed669cf8f3b6fc71359e4fb59b3f8c1d6bb6aa5a75fd1cb
Sep 08 14:06:05.699737 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:05.699693    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7" event={"ID":"fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70","Type":"ContainerStarted","Data":"8b5cd9bfd289f35eaa2ad6f23041798894bf1db685c9e48c10cb8be124d12ead"}
Sep 08 14:06:05.700131 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:05.699749    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7" event={"ID":"fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70","Type":"ContainerStarted","Data":"868177c7a172b7c3bed669cf8f3b6fc71359e4fb59b3f8c1d6bb6aa5a75fd1cb"}
Sep 08 14:06:05.700131 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:05.699794    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7"
Sep 08 14:06:05.718733 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:05.718656    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7" podStartSLOduration=1.7186392289999999 podStartE2EDuration="1.718639229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:06:05.716148102 +0000 UTC m=+2997.176792904" watchObservedRunningTime="2026-09-08 14:06:05.718639229 +0000 UTC m=+2997.179284039"
Sep 08 14:06:09.254124 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:09.254092    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 14:06:09.257574 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:09.257545    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 14:06:20.033492 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:20.033450    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h"]
Sep 08 14:06:26.141125 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:26.141079    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h" podUID="94c6e450-4c8b-47d1-adcd-68d2460c086d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:06:35.057230 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:35.057185    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h" podUID="94c6e450-4c8b-47d1-adcd-68d2460c086d" containerName="main" containerID="cri-o://24a18e7e64f5396c9be9220f26fc71809bbe67800e0ff4acb616fdeccd2a9083" gracePeriod=45
Sep 08 14:06:36.141130 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:36.141087    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h" podUID="94c6e450-4c8b-47d1-adcd-68d2460c086d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:06:36.706931 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:36.706900    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7"
Sep 08 14:06:46.141415 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:46.141370    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h" podUID="94c6e450-4c8b-47d1-adcd-68d2460c086d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:06:46.142148 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:46.141533    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h"
Sep 08 14:06:56.140941 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:06:56.140900    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h" podUID="94c6e450-4c8b-47d1-adcd-68d2460c086d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:07:05.318050 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:05.317985    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h"
Sep 08 14:07:05.424563 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:05.424529    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjfvs\" (UniqueName: \"kubernetes.io/projected/94c6e450-4c8b-47d1-adcd-68d2460c086d-kube-api-access-jjfvs\") pod \"94c6e450-4c8b-47d1-adcd-68d2460c086d\" (UID: \"94c6e450-4c8b-47d1-adcd-68d2460c086d\") "
Sep 08 14:07:05.424754 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:05.424587    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c6e450-4c8b-47d1-adcd-68d2460c086d-tls-certs\") pod \"94c6e450-4c8b-47d1-adcd-68d2460c086d\" (UID: \"94c6e450-4c8b-47d1-adcd-68d2460c086d\") "
Sep 08 14:07:05.426876 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:05.426847    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c6e450-4c8b-47d1-adcd-68d2460c086d-kube-api-access-jjfvs" (OuterVolumeSpecName: "kube-api-access-jjfvs") pod "94c6e450-4c8b-47d1-adcd-68d2460c086d" (UID: "94c6e450-4c8b-47d1-adcd-68d2460c086d"). InnerVolumeSpecName "kube-api-access-jjfvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:07:05.426929 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:05.426858    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94c6e450-4c8b-47d1-adcd-68d2460c086d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94c6e450-4c8b-47d1-adcd-68d2460c086d" (UID: "94c6e450-4c8b-47d1-adcd-68d2460c086d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:07:05.525536 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:05.525502    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjfvs\" (UniqueName: \"kubernetes.io/projected/94c6e450-4c8b-47d1-adcd-68d2460c086d-kube-api-access-jjfvs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:07:05.525536 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:05.525530    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c6e450-4c8b-47d1-adcd-68d2460c086d-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:07:06.016663 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:06.016623    2561 generic.go:356] "Generic (PLEG): container finished" podID="94c6e450-4c8b-47d1-adcd-68d2460c086d" containerID="24a18e7e64f5396c9be9220f26fc71809bbe67800e0ff4acb616fdeccd2a9083" exitCode=0
Sep 08 14:07:06.016974 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:06.016669    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h"
Sep 08 14:07:06.016974 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:06.016710    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h" event={"ID":"94c6e450-4c8b-47d1-adcd-68d2460c086d","Type":"ContainerDied","Data":"24a18e7e64f5396c9be9220f26fc71809bbe67800e0ff4acb616fdeccd2a9083"}
Sep 08 14:07:06.016974 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:06.016771    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h" event={"ID":"94c6e450-4c8b-47d1-adcd-68d2460c086d","Type":"ContainerDied","Data":"0863da69241be6dac6dd95d654ccd969973908c4723cbe8de752604d9b6bd1b9"}
Sep 08 14:07:06.016974 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:06.016802    2561 scope.go:117] "RemoveContainer" containerID="24a18e7e64f5396c9be9220f26fc71809bbe67800e0ff4acb616fdeccd2a9083"
Sep 08 14:07:06.026554 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:06.026533    2561 scope.go:117] "RemoveContainer" containerID="24a18e7e64f5396c9be9220f26fc71809bbe67800e0ff4acb616fdeccd2a9083"
Sep 08 14:07:06.026828 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:07:06.026809    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a18e7e64f5396c9be9220f26fc71809bbe67800e0ff4acb616fdeccd2a9083\": container with ID starting with 24a18e7e64f5396c9be9220f26fc71809bbe67800e0ff4acb616fdeccd2a9083 not found: ID does not exist" containerID="24a18e7e64f5396c9be9220f26fc71809bbe67800e0ff4acb616fdeccd2a9083"
Sep 08 14:07:06.026875 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:06.026837    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a18e7e64f5396c9be9220f26fc71809bbe67800e0ff4acb616fdeccd2a9083"} err="failed to get container status \"24a18e7e64f5396c9be9220f26fc71809bbe67800e0ff4acb616fdeccd2a9083\": rpc error: code = NotFound desc = could not find container \"24a18e7e64f5396c9be9220f26fc71809bbe67800e0ff4acb616fdeccd2a9083\": container with ID starting with 24a18e7e64f5396c9be9220f26fc71809bbe67800e0ff4acb616fdeccd2a9083 not found: ID does not exist"
Sep 08 14:07:06.038844 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:06.038801    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h"]
Sep 08 14:07:06.042386 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:06.042363    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schestd4h"]
Sep 08 14:07:07.007060 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:07.007026    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94c6e450-4c8b-47d1-adcd-68d2460c086d" path="/var/lib/kubelet/pods/94c6e450-4c8b-47d1-adcd-68d2460c086d/volumes"
Sep 08 14:07:16.703712 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:16.703668    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88"]
Sep 08 14:07:16.704240 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:16.704216    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94c6e450-4c8b-47d1-adcd-68d2460c086d" containerName="main"
Sep 08 14:07:16.704357 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:16.704243    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="94c6e450-4c8b-47d1-adcd-68d2460c086d" containerName="main"
Sep 08 14:07:16.704422 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:16.704383    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="94c6e450-4c8b-47d1-adcd-68d2460c086d" containerName="main"
Sep 08 14:07:16.709016 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:16.708991    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88"
Sep 08 14:07:16.711316 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:16.711281    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 08 14:07:16.712122 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:16.712102    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 08 14:07:16.712253 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:16.712223    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-9dd8p\""
Sep 08 14:07:16.712253 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:16.712238    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 08 14:07:16.712464 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:16.712249    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-pn9kc\""
Sep 08 14:07:16.718439 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:16.718412    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88"]
Sep 08 14:07:16.722075 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:16.722049    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9776f2ce-c34e-4855-8571-5f6c1c295cfc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88\" (UID: \"9776f2ce-c34e-4855-8571-5f6c1c295cfc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88"
Sep 08 14:07:16.722183 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:16.722093    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x288\" (UniqueName: \"kubernetes.io/projected/9776f2ce-c34e-4855-8571-5f6c1c295cfc-kube-api-access-5x288\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88\" (UID: \"9776f2ce-c34e-4855-8571-5f6c1c295cfc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88"
Sep 08 14:07:16.822929 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:16.822886    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9776f2ce-c34e-4855-8571-5f6c1c295cfc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88\" (UID: \"9776f2ce-c34e-4855-8571-5f6c1c295cfc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88"
Sep 08 14:07:16.822929 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:16.822930    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x288\" (UniqueName: \"kubernetes.io/projected/9776f2ce-c34e-4855-8571-5f6c1c295cfc-kube-api-access-5x288\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88\" (UID: \"9776f2ce-c34e-4855-8571-5f6c1c295cfc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88"
Sep 08 14:07:16.825500 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:16.825474    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9776f2ce-c34e-4855-8571-5f6c1c295cfc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88\" (UID: \"9776f2ce-c34e-4855-8571-5f6c1c295cfc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88"
Sep 08 14:07:16.831902 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:16.831874    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x288\" (UniqueName: \"kubernetes.io/projected/9776f2ce-c34e-4855-8571-5f6c1c295cfc-kube-api-access-5x288\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88\" (UID: \"9776f2ce-c34e-4855-8571-5f6c1c295cfc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88"
Sep 08 14:07:17.020345 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:17.020245    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88"
Sep 08 14:07:17.305057 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:17.305026    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88"]
Sep 08 14:07:17.306992 ip-10-0-130-178 kubenswrapper[2561]: W0908 14:07:17.306965    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9776f2ce_c34e_4855_8571_5f6c1c295cfc.slice/crio-d5f2e8ebd9ce1ebd11b99e6f4e3693f1cb9e9ea7dbc064abca41484b24a370a3 WatchSource:0}: Error finding container d5f2e8ebd9ce1ebd11b99e6f4e3693f1cb9e9ea7dbc064abca41484b24a370a3: Status 404 returned error can't find the container with id d5f2e8ebd9ce1ebd11b99e6f4e3693f1cb9e9ea7dbc064abca41484b24a370a3
Sep 08 14:07:18.084583 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:18.084547    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88" event={"ID":"9776f2ce-c34e-4855-8571-5f6c1c295cfc","Type":"ContainerStarted","Data":"d2f756d31569a4d9c6d71095bbaa1e570b50a906d5c96a1acf144f7287e871c0"}
Sep 08 14:07:18.084979 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:18.084605    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88" event={"ID":"9776f2ce-c34e-4855-8571-5f6c1c295cfc","Type":"ContainerStarted","Data":"d5f2e8ebd9ce1ebd11b99e6f4e3693f1cb9e9ea7dbc064abca41484b24a370a3"}
Sep 08 14:07:18.084979 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:18.084632    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88"
Sep 08 14:07:18.111237 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:18.111164    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88" podStartSLOduration=2.111142372 podStartE2EDuration="2.111142372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:07:18.104835242 +0000 UTC m=+3069.565480052" watchObservedRunningTime="2026-09-08 14:07:18.111142372 +0000 UTC m=+3069.571787182"
Sep 08 14:07:49.092378 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:07:49.092340    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88"
Sep 08 14:08:19.584325 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:08:19.584255    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7"]
Sep 08 14:08:26.707742 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:08:26.707696    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7" podUID="fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:08:34.605233 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:08:34.605139    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7" podUID="fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70" containerName="main" containerID="cri-o://8b5cd9bfd289f35eaa2ad6f23041798894bf1db685c9e48c10cb8be124d12ead" gracePeriod=45
Sep 08 14:08:36.707120 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:08:36.707082    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7" podUID="fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:08:38.356339 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:08:38.356297    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88"]
Sep 08 14:08:39.092071 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:08:39.092032    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88" podUID="9776f2ce-c34e-4855-8571-5f6c1c295cfc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:08:46.707146 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:08:46.707101    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7" podUID="fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:08:46.707524 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:08:46.707202    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7"
Sep 08 14:08:49.092416 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:08:49.092372    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88" podUID="9776f2ce-c34e-4855-8571-5f6c1c295cfc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:08:53.380041 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:08:53.379997    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88" podUID="9776f2ce-c34e-4855-8571-5f6c1c295cfc" containerName="main" containerID="cri-o://d2f756d31569a4d9c6d71095bbaa1e570b50a906d5c96a1acf144f7287e871c0" gracePeriod=45
Sep 08 14:08:56.707385 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:08:56.707345    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7" podUID="fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:08:59.091716 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:08:59.091678    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88" podUID="9776f2ce-c34e-4855-8571-5f6c1c295cfc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:08:59.092099 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:08:59.091790    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88"
Sep 08 14:09:04.856231 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:04.856202    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7"
Sep 08 14:09:04.930558 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:04.930453    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70-tls-certs\") pod \"fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70\" (UID: \"fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70\") "
Sep 08 14:09:04.930558 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:04.930556    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdnd7\" (UniqueName: \"kubernetes.io/projected/fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70-kube-api-access-kdnd7\") pod \"fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70\" (UID: \"fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70\") "
Sep 08 14:09:04.932943 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:04.932910    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70-kube-api-access-kdnd7" (OuterVolumeSpecName: "kube-api-access-kdnd7") pod "fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70" (UID: "fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70"). InnerVolumeSpecName "kube-api-access-kdnd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:09:04.933091 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:04.932911    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70" (UID: "fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:09:05.032050 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:05.032013    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdnd7\" (UniqueName: \"kubernetes.io/projected/fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70-kube-api-access-kdnd7\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:09:05.032050 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:05.032050    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:09:05.700828 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:05.700791    2561 generic.go:356] "Generic (PLEG): container finished" podID="fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70" containerID="8b5cd9bfd289f35eaa2ad6f23041798894bf1db685c9e48c10cb8be124d12ead" exitCode=0
Sep 08 14:09:05.701009 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:05.700841    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7"
Sep 08 14:09:05.701009 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:05.700879    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7" event={"ID":"fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70","Type":"ContainerDied","Data":"8b5cd9bfd289f35eaa2ad6f23041798894bf1db685c9e48c10cb8be124d12ead"}
Sep 08 14:09:05.701009 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:05.700940    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7" event={"ID":"fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70","Type":"ContainerDied","Data":"868177c7a172b7c3bed669cf8f3b6fc71359e4fb59b3f8c1d6bb6aa5a75fd1cb"}
Sep 08 14:09:05.701009 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:05.700969    2561 scope.go:117] "RemoveContainer" containerID="8b5cd9bfd289f35eaa2ad6f23041798894bf1db685c9e48c10cb8be124d12ead"
Sep 08 14:09:05.710071 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:05.710043    2561 scope.go:117] "RemoveContainer" containerID="8b5cd9bfd289f35eaa2ad6f23041798894bf1db685c9e48c10cb8be124d12ead"
Sep 08 14:09:05.710378 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:09:05.710351    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b5cd9bfd289f35eaa2ad6f23041798894bf1db685c9e48c10cb8be124d12ead\": container with ID starting with 8b5cd9bfd289f35eaa2ad6f23041798894bf1db685c9e48c10cb8be124d12ead not found: ID does not exist" containerID="8b5cd9bfd289f35eaa2ad6f23041798894bf1db685c9e48c10cb8be124d12ead"
Sep 08 14:09:05.710494 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:05.710386    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5cd9bfd289f35eaa2ad6f23041798894bf1db685c9e48c10cb8be124d12ead"} err="failed to get container status \"8b5cd9bfd289f35eaa2ad6f23041798894bf1db685c9e48c10cb8be124d12ead\": rpc error: code = NotFound desc = could not find container \"8b5cd9bfd289f35eaa2ad6f23041798894bf1db685c9e48c10cb8be124d12ead\": container with ID starting with 8b5cd9bfd289f35eaa2ad6f23041798894bf1db685c9e48c10cb8be124d12ead not found: ID does not exist"
Sep 08 14:09:05.718608 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:05.718577    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7"]
Sep 08 14:09:05.721641 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:05.721617    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheq8lh7"]
Sep 08 14:09:07.010324 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:07.010265    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70" path="/var/lib/kubelet/pods/fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70/volumes"
Sep 08 14:09:09.091564 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:09.091522    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88" podUID="9776f2ce-c34e-4855-8571-5f6c1c295cfc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:09:19.091822 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:19.091783    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88" podUID="9776f2ce-c34e-4855-8571-5f6c1c295cfc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:09:23.632394 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:23.632366    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88"
Sep 08 14:09:23.691540 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:23.691509    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5x288\" (UniqueName: \"kubernetes.io/projected/9776f2ce-c34e-4855-8571-5f6c1c295cfc-kube-api-access-5x288\") pod \"9776f2ce-c34e-4855-8571-5f6c1c295cfc\" (UID: \"9776f2ce-c34e-4855-8571-5f6c1c295cfc\") "
Sep 08 14:09:23.691722 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:23.691560    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9776f2ce-c34e-4855-8571-5f6c1c295cfc-tls-certs\") pod \"9776f2ce-c34e-4855-8571-5f6c1c295cfc\" (UID: \"9776f2ce-c34e-4855-8571-5f6c1c295cfc\") "
Sep 08 14:09:23.693725 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:23.693693    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9776f2ce-c34e-4855-8571-5f6c1c295cfc-kube-api-access-5x288" (OuterVolumeSpecName: "kube-api-access-5x288") pod "9776f2ce-c34e-4855-8571-5f6c1c295cfc" (UID: "9776f2ce-c34e-4855-8571-5f6c1c295cfc"). InnerVolumeSpecName "kube-api-access-5x288". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:09:23.693855 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:23.693765    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9776f2ce-c34e-4855-8571-5f6c1c295cfc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9776f2ce-c34e-4855-8571-5f6c1c295cfc" (UID: "9776f2ce-c34e-4855-8571-5f6c1c295cfc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:09:23.792627 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:23.792587    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x288\" (UniqueName: \"kubernetes.io/projected/9776f2ce-c34e-4855-8571-5f6c1c295cfc-kube-api-access-5x288\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:09:23.792627 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:23.792620    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9776f2ce-c34e-4855-8571-5f6c1c295cfc-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:09:23.800433 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:23.800398    2561 generic.go:356] "Generic (PLEG): container finished" podID="9776f2ce-c34e-4855-8571-5f6c1c295cfc" containerID="d2f756d31569a4d9c6d71095bbaa1e570b50a906d5c96a1acf144f7287e871c0" exitCode=0
Sep 08 14:09:23.800618 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:23.800454    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88"
Sep 08 14:09:23.800618 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:23.800495    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88" event={"ID":"9776f2ce-c34e-4855-8571-5f6c1c295cfc","Type":"ContainerDied","Data":"d2f756d31569a4d9c6d71095bbaa1e570b50a906d5c96a1acf144f7287e871c0"}
Sep 08 14:09:23.800618 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:23.800565    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88" event={"ID":"9776f2ce-c34e-4855-8571-5f6c1c295cfc","Type":"ContainerDied","Data":"d5f2e8ebd9ce1ebd11b99e6f4e3693f1cb9e9ea7dbc064abca41484b24a370a3"}
Sep 08 14:09:23.800618 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:23.800597    2561 scope.go:117] "RemoveContainer" containerID="d2f756d31569a4d9c6d71095bbaa1e570b50a906d5c96a1acf144f7287e871c0"
Sep 08 14:09:23.809862 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:23.809843    2561 scope.go:117] "RemoveContainer" containerID="d2f756d31569a4d9c6d71095bbaa1e570b50a906d5c96a1acf144f7287e871c0"
Sep 08 14:09:23.810133 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:09:23.810115    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f756d31569a4d9c6d71095bbaa1e570b50a906d5c96a1acf144f7287e871c0\": container with ID starting with d2f756d31569a4d9c6d71095bbaa1e570b50a906d5c96a1acf144f7287e871c0 not found: ID does not exist" containerID="d2f756d31569a4d9c6d71095bbaa1e570b50a906d5c96a1acf144f7287e871c0"
Sep 08 14:09:23.810192 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:23.810141    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f756d31569a4d9c6d71095bbaa1e570b50a906d5c96a1acf144f7287e871c0"} err="failed to get container status \"d2f756d31569a4d9c6d71095bbaa1e570b50a906d5c96a1acf144f7287e871c0\": rpc error: code = NotFound desc = could not find container \"d2f756d31569a4d9c6d71095bbaa1e570b50a906d5c96a1acf144f7287e871c0\": container with ID starting with d2f756d31569a4d9c6d71095bbaa1e570b50a906d5c96a1acf144f7287e871c0 not found: ID does not exist"
Sep 08 14:09:23.823802 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:23.823778    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88"]
Sep 08 14:09:23.830420 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:23.830397    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche84m88"]
Sep 08 14:09:25.006760 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:25.006723    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9776f2ce-c34e-4855-8571-5f6c1c295cfc" path="/var/lib/kubelet/pods/9776f2ce-c34e-4855-8571-5f6c1c295cfc/volumes"
Sep 08 14:09:37.960120 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:37.960085    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp"]
Sep 08 14:09:37.960640 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:37.960618    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70" containerName="main"
Sep 08 14:09:37.960724 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:37.960643    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70" containerName="main"
Sep 08 14:09:37.960724 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:37.960657    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9776f2ce-c34e-4855-8571-5f6c1c295cfc" containerName="main"
Sep 08 14:09:37.960724 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:37.960665    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9776f2ce-c34e-4855-8571-5f6c1c295cfc" containerName="main"
Sep 08 14:09:37.960880 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:37.960760    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcd07d7f-4d7e-46f7-8f85-e473cfaa9f70" containerName="main"
Sep 08 14:09:37.960880 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:37.960771    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9776f2ce-c34e-4855-8571-5f6c1c295cfc" containerName="main"
Sep 08 14:09:37.965440 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:37.965413    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp"
Sep 08 14:09:37.968904 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:37.968879    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-dl8fm\""
Sep 08 14:09:37.969033 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:37.968890    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 08 14:09:37.969033 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:37.968953    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 08 14:09:37.969155 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:37.969026    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-wmq8d\""
Sep 08 14:09:37.969155 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:37.969090    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 08 14:09:37.982469 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:37.982437    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp"]
Sep 08 14:09:38.019867 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:38.019826    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/064bf465-5495-4c73-acaa-d6bf28224487-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp\" (UID: \"064bf465-5495-4c73-acaa-d6bf28224487\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp"
Sep 08 14:09:38.020072 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:38.019955    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gccwz\" (UniqueName: \"kubernetes.io/projected/064bf465-5495-4c73-acaa-d6bf28224487-kube-api-access-gccwz\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp\" (UID: \"064bf465-5495-4c73-acaa-d6bf28224487\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp"
Sep 08 14:09:38.121426 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:38.121385    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gccwz\" (UniqueName: \"kubernetes.io/projected/064bf465-5495-4c73-acaa-d6bf28224487-kube-api-access-gccwz\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp\" (UID: \"064bf465-5495-4c73-acaa-d6bf28224487\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp"
Sep 08 14:09:38.121637 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:38.121536    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/064bf465-5495-4c73-acaa-d6bf28224487-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp\" (UID: \"064bf465-5495-4c73-acaa-d6bf28224487\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp"
Sep 08 14:09:38.124086 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:38.124061    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/064bf465-5495-4c73-acaa-d6bf28224487-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp\" (UID: \"064bf465-5495-4c73-acaa-d6bf28224487\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp"
Sep 08 14:09:38.130003 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:38.129970    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gccwz\" (UniqueName: \"kubernetes.io/projected/064bf465-5495-4c73-acaa-d6bf28224487-kube-api-access-gccwz\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp\" (UID: \"064bf465-5495-4c73-acaa-d6bf28224487\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp"
Sep 08 14:09:38.276362 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:38.276261    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp"
Sep 08 14:09:38.555463 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:38.555433    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp"]
Sep 08 14:09:38.557153 ip-10-0-130-178 kubenswrapper[2561]: W0908 14:09:38.557123    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064bf465_5495_4c73_acaa_d6bf28224487.slice/crio-58e8189beba208ba1672431c8cf2473cccec6b4f7575711f1a3c0b1a4ea9e9e8 WatchSource:0}: Error finding container 58e8189beba208ba1672431c8cf2473cccec6b4f7575711f1a3c0b1a4ea9e9e8: Status 404 returned error can't find the container with id 58e8189beba208ba1672431c8cf2473cccec6b4f7575711f1a3c0b1a4ea9e9e8
Sep 08 14:09:38.881512 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:38.881409    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp" event={"ID":"064bf465-5495-4c73-acaa-d6bf28224487","Type":"ContainerStarted","Data":"f019cdf0526ff249116e56ac3048dcfcce3115d3e2491c4f25fcce70e6062f4c"}
Sep 08 14:09:38.881512 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:38.881475    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp"
Sep 08 14:09:38.881512 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:38.881495    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp" event={"ID":"064bf465-5495-4c73-acaa-d6bf28224487","Type":"ContainerStarted","Data":"58e8189beba208ba1672431c8cf2473cccec6b4f7575711f1a3c0b1a4ea9e9e8"}
Sep 08 14:09:38.901275 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:09:38.901216    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp" podStartSLOduration=1.901197678 podStartE2EDuration="1.901197678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:09:38.899764268 +0000 UTC m=+3210.360409077" watchObservedRunningTime="2026-09-08 14:09:38.901197678 +0000 UTC m=+3210.361842490"
Sep 08 14:10:05.643177 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:05.643084    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb"]
Sep 08 14:10:05.646911 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:05.646882    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb"
Sep 08 14:10:05.650304 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:05.650265    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 08 14:10:05.650644 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:05.650333    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 08 14:10:05.650758 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:05.650454    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-wgjrx\""
Sep 08 14:10:05.650758 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:05.650504    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-jmqm4\""
Sep 08 14:10:05.650878 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:05.650515    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 08 14:10:05.661710 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:05.661678    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb"]
Sep 08 14:10:05.775452 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:05.775410    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfzql\" (UniqueName: \"kubernetes.io/projected/8ea432a3-22c4-4cea-9b61-07fd6c842b0e-kube-api-access-sfzql\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb\" (UID: \"8ea432a3-22c4-4cea-9b61-07fd6c842b0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb"
Sep 08 14:10:05.775641 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:05.775472    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea432a3-22c4-4cea-9b61-07fd6c842b0e-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb\" (UID: \"8ea432a3-22c4-4cea-9b61-07fd6c842b0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb"
Sep 08 14:10:05.876456 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:05.876420    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfzql\" (UniqueName: \"kubernetes.io/projected/8ea432a3-22c4-4cea-9b61-07fd6c842b0e-kube-api-access-sfzql\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb\" (UID: \"8ea432a3-22c4-4cea-9b61-07fd6c842b0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb"
Sep 08 14:10:05.876648 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:05.876474    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea432a3-22c4-4cea-9b61-07fd6c842b0e-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb\" (UID: \"8ea432a3-22c4-4cea-9b61-07fd6c842b0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb"
Sep 08 14:10:05.878988 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:05.878959    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea432a3-22c4-4cea-9b61-07fd6c842b0e-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb\" (UID: \"8ea432a3-22c4-4cea-9b61-07fd6c842b0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb"
Sep 08 14:10:05.885314 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:05.885259    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfzql\" (UniqueName: \"kubernetes.io/projected/8ea432a3-22c4-4cea-9b61-07fd6c842b0e-kube-api-access-sfzql\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb\" (UID: \"8ea432a3-22c4-4cea-9b61-07fd6c842b0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb"
Sep 08 14:10:05.960216 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:05.960176    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb"
Sep 08 14:10:06.238694 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:06.238665    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb"]
Sep 08 14:10:06.240799 ip-10-0-130-178 kubenswrapper[2561]: W0908 14:10:06.240766    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea432a3_22c4_4cea_9b61_07fd6c842b0e.slice/crio-6d76130398e685d08c38aedecc43b47071f6fdf36c92fd1056294f91753180b1 WatchSource:0}: Error finding container 6d76130398e685d08c38aedecc43b47071f6fdf36c92fd1056294f91753180b1: Status 404 returned error can't find the container with id 6d76130398e685d08c38aedecc43b47071f6fdf36c92fd1056294f91753180b1
Sep 08 14:10:07.039170 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:07.039126    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb" event={"ID":"8ea432a3-22c4-4cea-9b61-07fd6c842b0e","Type":"ContainerStarted","Data":"62f86daa82ea6cdd5f99727ab46779d2ecb793ea1fee275cfd6e53284259a80e"}
Sep 08 14:10:07.039654 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:07.039195    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb" event={"ID":"8ea432a3-22c4-4cea-9b61-07fd6c842b0e","Type":"ContainerStarted","Data":"6d76130398e685d08c38aedecc43b47071f6fdf36c92fd1056294f91753180b1"}
Sep 08 14:10:07.039654 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:07.039252    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb"
Sep 08 14:10:07.060054 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:07.060000    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb" podStartSLOduration=2.059983364 podStartE2EDuration="2.059983364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:10:07.057688928 +0000 UTC m=+3238.518333738" watchObservedRunningTime="2026-09-08 14:10:07.059983364 +0000 UTC m=+3238.520628173"
Sep 08 14:10:09.894006 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:09.888664    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp"
Sep 08 14:10:38.047065 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:10:38.047026    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb"
Sep 08 14:11:09.288625 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:11:09.288593    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 14:11:09.295434 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:11:09.295406    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 14:11:42.521166 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:11:42.521074    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp"]
Sep 08 14:11:49.888002 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:11:49.887961    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp" podUID="064bf465-5495-4c73-acaa-d6bf28224487" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:11:57.541926 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:11:57.541877    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp" podUID="064bf465-5495-4c73-acaa-d6bf28224487" containerName="main" containerID="cri-o://f019cdf0526ff249116e56ac3048dcfcce3115d3e2491c4f25fcce70e6062f4c" gracePeriod=45
Sep 08 14:11:59.887561 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:11:59.887515    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp" podUID="064bf465-5495-4c73-acaa-d6bf28224487" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:12:07.480269 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:07.480226    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb"]
Sep 08 14:12:08.047484 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:08.047447    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb" podUID="8ea432a3-22c4-4cea-9b61-07fd6c842b0e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:12:09.888012 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:09.887954    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp" podUID="064bf465-5495-4c73-acaa-d6bf28224487" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:12:09.888484 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:09.888073    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp"
Sep 08 14:12:18.047025 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:18.046984    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb" podUID="8ea432a3-22c4-4cea-9b61-07fd6c842b0e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:12:19.887567 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:19.887527    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp" podUID="064bf465-5495-4c73-acaa-d6bf28224487" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:12:22.500950 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:22.500878    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb" podUID="8ea432a3-22c4-4cea-9b61-07fd6c842b0e" containerName="main" containerID="cri-o://62f86daa82ea6cdd5f99727ab46779d2ecb793ea1fee275cfd6e53284259a80e" gracePeriod=45
Sep 08 14:12:27.791723 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:27.791690    2561 generic.go:356] "Generic (PLEG): container finished" podID="064bf465-5495-4c73-acaa-d6bf28224487" containerID="f019cdf0526ff249116e56ac3048dcfcce3115d3e2491c4f25fcce70e6062f4c" exitCode=0
Sep 08 14:12:27.792132 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:27.791761    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp" event={"ID":"064bf465-5495-4c73-acaa-d6bf28224487","Type":"ContainerDied","Data":"f019cdf0526ff249116e56ac3048dcfcce3115d3e2491c4f25fcce70e6062f4c"}
Sep 08 14:12:27.792132 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:27.791811    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp" event={"ID":"064bf465-5495-4c73-acaa-d6bf28224487","Type":"ContainerDied","Data":"58e8189beba208ba1672431c8cf2473cccec6b4f7575711f1a3c0b1a4ea9e9e8"}
Sep 08 14:12:27.792132 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:27.791827    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58e8189beba208ba1672431c8cf2473cccec6b4f7575711f1a3c0b1a4ea9e9e8"
Sep 08 14:12:27.799328 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:27.799272    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp"
Sep 08 14:12:27.907124 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:27.907031    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/064bf465-5495-4c73-acaa-d6bf28224487-tls-certs\") pod \"064bf465-5495-4c73-acaa-d6bf28224487\" (UID: \"064bf465-5495-4c73-acaa-d6bf28224487\") "
Sep 08 14:12:27.907274 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:27.907158    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gccwz\" (UniqueName: \"kubernetes.io/projected/064bf465-5495-4c73-acaa-d6bf28224487-kube-api-access-gccwz\") pod \"064bf465-5495-4c73-acaa-d6bf28224487\" (UID: \"064bf465-5495-4c73-acaa-d6bf28224487\") "
Sep 08 14:12:27.909387 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:27.909360    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/064bf465-5495-4c73-acaa-d6bf28224487-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "064bf465-5495-4c73-acaa-d6bf28224487" (UID: "064bf465-5495-4c73-acaa-d6bf28224487"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:12:27.909521 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:27.909406    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/064bf465-5495-4c73-acaa-d6bf28224487-kube-api-access-gccwz" (OuterVolumeSpecName: "kube-api-access-gccwz") pod "064bf465-5495-4c73-acaa-d6bf28224487" (UID: "064bf465-5495-4c73-acaa-d6bf28224487"). InnerVolumeSpecName "kube-api-access-gccwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:12:28.008611 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:28.008575    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/064bf465-5495-4c73-acaa-d6bf28224487-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:12:28.008611 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:28.008606    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gccwz\" (UniqueName: \"kubernetes.io/projected/064bf465-5495-4c73-acaa-d6bf28224487-kube-api-access-gccwz\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:12:28.047860 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:28.047817    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb" podUID="8ea432a3-22c4-4cea-9b61-07fd6c842b0e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:12:28.048022 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:28.047933    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb"
Sep 08 14:12:28.796916 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:28.796884    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp"
Sep 08 14:12:28.818939 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:28.818909    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp"]
Sep 08 14:12:28.822654 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:28.822631    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc584gnrp"]
Sep 08 14:12:29.007273 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:29.007238    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="064bf465-5495-4c73-acaa-d6bf28224487" path="/var/lib/kubelet/pods/064bf465-5495-4c73-acaa-d6bf28224487/volumes"
Sep 08 14:12:32.666595 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.666553    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"]
Sep 08 14:12:32.667114 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.667092    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="064bf465-5495-4c73-acaa-d6bf28224487" containerName="main"
Sep 08 14:12:32.667182 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.667117    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="064bf465-5495-4c73-acaa-d6bf28224487" containerName="main"
Sep 08 14:12:32.667246 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.667240    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="064bf465-5495-4c73-acaa-d6bf28224487" containerName="main"
Sep 08 14:12:32.671007 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.670982    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.674510 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.674483    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-rbb56\""
Sep 08 14:12:32.674658 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.674524    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 08 14:12:32.674724 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.674681    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 08 14:12:32.674785 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.674766    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 08 14:12:32.683255 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.683232    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"]
Sep 08 14:12:32.751794 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.751755    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.751992 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.751852    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.751992 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.751881    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.751992 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.751903    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmx6v\" (UniqueName: \"kubernetes.io/projected/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-kube-api-access-cmx6v\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.751992 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.751928    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.752202 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.752034    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.752202 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.752082    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.752202 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.752171    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.752202 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.752197    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.852979 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.852936    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.852979 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.852980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.853234 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.852996    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmx6v\" (UniqueName: \"kubernetes.io/projected/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-kube-api-access-cmx6v\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.853234 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.853017    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.853234 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.853048    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.853234 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.853077    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.853234 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.853120    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.853234 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.853143    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.853234 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.853218    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.853645 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.853514    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.853645 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.853602    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.853734 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.853642    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.853734 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.853677    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.853867 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.853848    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.855528 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.855507    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.855665 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.855650    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.874783 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.874752    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmx6v\" (UniqueName: \"kubernetes.io/projected/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-kube-api-access-cmx6v\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.875137 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.875120    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-2c9jv\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:32.984249 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:32.984211    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:33.267359 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:33.267327    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"]
Sep 08 14:12:33.267908 ip-10-0-130-178 kubenswrapper[2561]: W0908 14:12:33.267882    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b39f0f_d17f_4cdb_a61a_44c628dd7d23.slice/crio-853ee11f7857d7f0675ec1a9492f8e0c47f726a8717e68b40445236cadfa312d WatchSource:0}: Error finding container 853ee11f7857d7f0675ec1a9492f8e0c47f726a8717e68b40445236cadfa312d: Status 404 returned error can't find the container with id 853ee11f7857d7f0675ec1a9492f8e0c47f726a8717e68b40445236cadfa312d
Sep 08 14:12:33.269869 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:33.269835    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 14:12:33.269956 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:33.269916    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 14:12:33.269956 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:33.269944    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 14:12:33.825098 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:33.825059    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv" event={"ID":"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23","Type":"ContainerStarted","Data":"9661a365253e11b9269e67bb789b7cdcf14a8fd33628b3051010a394e6c6f1ab"}
Sep 08 14:12:33.825514 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:33.825134    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv" event={"ID":"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23","Type":"ContainerStarted","Data":"853ee11f7857d7f0675ec1a9492f8e0c47f726a8717e68b40445236cadfa312d"}
Sep 08 14:12:33.846753 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:33.846668    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv" podStartSLOduration=1.846649224 podStartE2EDuration="1.846649224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:12:33.845694799 +0000 UTC m=+3385.306339608" watchObservedRunningTime="2026-09-08 14:12:33.846649224 +0000 UTC m=+3385.307294035"
Sep 08 14:12:33.984908 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:33.984870    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:34.989133 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:34.989101    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:35.835880 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:35.835840    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:35.836835 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:35.836811    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:38.047766 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:38.047715    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb" podUID="8ea432a3-22c4-4cea-9b61-07fd6c842b0e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:12:45.901741 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:45.901708    2561 generic.go:356] "Generic (PLEG): container finished" podID="d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" containerID="9661a365253e11b9269e67bb789b7cdcf14a8fd33628b3051010a394e6c6f1ab" exitCode=0
Sep 08 14:12:45.902170 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:45.901785    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv" event={"ID":"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23","Type":"ContainerDied","Data":"9661a365253e11b9269e67bb789b7cdcf14a8fd33628b3051010a394e6c6f1ab"}
Sep 08 14:12:45.902220 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:45.902199    2561 scope.go:117] "RemoveContainer" containerID="9661a365253e11b9269e67bb789b7cdcf14a8fd33628b3051010a394e6c6f1ab"
Sep 08 14:12:45.903220 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:45.903195    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 14:12:45.903305 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:45.903259    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 14:12:45.903358 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:45.903314    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 14:12:46.909115 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:46.909084    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv" event={"ID":"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23","Type":"ContainerStarted","Data":"aff28679859364143d044153464b356acd3298cada86c69463403cc8f51774b8"}
Sep 08 14:12:46.985106 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:46.985061    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:46.986105 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:46.986070    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv" podUID="d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 08 14:12:47.690440 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:47.690393    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm"]
Sep 08 14:12:47.692840 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:47.692816    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm"
Sep 08 14:12:47.695275 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:47.695251    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-4r66n\""
Sep 08 14:12:47.695436 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:47.695252    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 08 14:12:47.695436 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:47.695252    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-dx8h5\""
Sep 08 14:12:47.709794 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:47.709765    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm"]
Sep 08 14:12:47.800135 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:47.800095    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbjns\" (UniqueName: \"kubernetes.io/projected/ef354a2c-58ea-41e8-b76d-15a46e8f8999-kube-api-access-kbjns\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm\" (UID: \"ef354a2c-58ea-41e8-b76d-15a46e8f8999\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm"
Sep 08 14:12:47.800354 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:47.800163    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef354a2c-58ea-41e8-b76d-15a46e8f8999-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm\" (UID: \"ef354a2c-58ea-41e8-b76d-15a46e8f8999\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm"
Sep 08 14:12:47.900846 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:47.900804    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbjns\" (UniqueName: \"kubernetes.io/projected/ef354a2c-58ea-41e8-b76d-15a46e8f8999-kube-api-access-kbjns\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm\" (UID: \"ef354a2c-58ea-41e8-b76d-15a46e8f8999\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm"
Sep 08 14:12:47.901064 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:47.900901    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef354a2c-58ea-41e8-b76d-15a46e8f8999-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm\" (UID: \"ef354a2c-58ea-41e8-b76d-15a46e8f8999\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm"
Sep 08 14:12:47.903693 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:47.903664    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef354a2c-58ea-41e8-b76d-15a46e8f8999-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm\" (UID: \"ef354a2c-58ea-41e8-b76d-15a46e8f8999\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm"
Sep 08 14:12:47.909981 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:47.909953    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbjns\" (UniqueName: \"kubernetes.io/projected/ef354a2c-58ea-41e8-b76d-15a46e8f8999-kube-api-access-kbjns\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm\" (UID: \"ef354a2c-58ea-41e8-b76d-15a46e8f8999\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm"
Sep 08 14:12:47.984967 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:47.984870    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv" podUID="d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 08 14:12:48.004937 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:48.004905    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm"
Sep 08 14:12:48.066877 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:48.066410    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb" podUID="8ea432a3-22c4-4cea-9b61-07fd6c842b0e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:12:48.305828 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:48.305801    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm"]
Sep 08 14:12:48.307662 ip-10-0-130-178 kubenswrapper[2561]: W0908 14:12:48.307628    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef354a2c_58ea_41e8_b76d_15a46e8f8999.slice/crio-471cce2d7520961b72e708340f9a6990352779d8edefef6bddb2ee54cdde562d WatchSource:0}: Error finding container 471cce2d7520961b72e708340f9a6990352779d8edefef6bddb2ee54cdde562d: Status 404 returned error can't find the container with id 471cce2d7520961b72e708340f9a6990352779d8edefef6bddb2ee54cdde562d
Sep 08 14:12:48.922230 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:48.922193    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm" event={"ID":"ef354a2c-58ea-41e8-b76d-15a46e8f8999","Type":"ContainerStarted","Data":"39ccd7262ae55343175f0e8689e0de4dbc6400993f5a136bf6a47abdfe4f05af"}
Sep 08 14:12:48.922640 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:48.922273    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm" event={"ID":"ef354a2c-58ea-41e8-b76d-15a46e8f8999","Type":"ContainerStarted","Data":"471cce2d7520961b72e708340f9a6990352779d8edefef6bddb2ee54cdde562d"}
Sep 08 14:12:48.922640 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:48.922319    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm"
Sep 08 14:12:48.939221 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:48.939166    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm" podStartSLOduration=1.939150033 podStartE2EDuration="1.939150033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:12:48.938259307 +0000 UTC m=+3400.398904121" watchObservedRunningTime="2026-09-08 14:12:48.939150033 +0000 UTC m=+3400.399794842"
Sep 08 14:12:48.984594 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:48.984553    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv" podUID="d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 08 14:12:49.985274 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:49.985231    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv" podUID="d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 08 14:12:50.836104 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:50.836065    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:50.985643 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:50.985598    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv" podUID="d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 08 14:12:52.052968 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:52.052921    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:52.054862 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:52.054831    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:12:52.751626 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:52.751595    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb"
Sep 08 14:12:52.848378 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:52.848251    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea432a3-22c4-4cea-9b61-07fd6c842b0e-tls-certs\") pod \"8ea432a3-22c4-4cea-9b61-07fd6c842b0e\" (UID: \"8ea432a3-22c4-4cea-9b61-07fd6c842b0e\") "
Sep 08 14:12:52.848378 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:52.848356    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfzql\" (UniqueName: \"kubernetes.io/projected/8ea432a3-22c4-4cea-9b61-07fd6c842b0e-kube-api-access-sfzql\") pod \"8ea432a3-22c4-4cea-9b61-07fd6c842b0e\" (UID: \"8ea432a3-22c4-4cea-9b61-07fd6c842b0e\") "
Sep 08 14:12:52.850580 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:52.850549    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea432a3-22c4-4cea-9b61-07fd6c842b0e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ea432a3-22c4-4cea-9b61-07fd6c842b0e" (UID: "8ea432a3-22c4-4cea-9b61-07fd6c842b0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:12:52.850674 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:52.850601    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea432a3-22c4-4cea-9b61-07fd6c842b0e-kube-api-access-sfzql" (OuterVolumeSpecName: "kube-api-access-sfzql") pod "8ea432a3-22c4-4cea-9b61-07fd6c842b0e" (UID: "8ea432a3-22c4-4cea-9b61-07fd6c842b0e"). InnerVolumeSpecName "kube-api-access-sfzql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:12:52.949097 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:52.949063    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ea432a3-22c4-4cea-9b61-07fd6c842b0e-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:12:52.949097 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:52.949097    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfzql\" (UniqueName: \"kubernetes.io/projected/8ea432a3-22c4-4cea-9b61-07fd6c842b0e-kube-api-access-sfzql\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:12:52.949938 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:52.949913    2561 generic.go:356] "Generic (PLEG): container finished" podID="8ea432a3-22c4-4cea-9b61-07fd6c842b0e" containerID="62f86daa82ea6cdd5f99727ab46779d2ecb793ea1fee275cfd6e53284259a80e" exitCode=0
Sep 08 14:12:52.950033 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:52.949956    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb"
Sep 08 14:12:52.950033 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:52.949993    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb" event={"ID":"8ea432a3-22c4-4cea-9b61-07fd6c842b0e","Type":"ContainerDied","Data":"62f86daa82ea6cdd5f99727ab46779d2ecb793ea1fee275cfd6e53284259a80e"}
Sep 08 14:12:52.950117 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:52.950033    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb" event={"ID":"8ea432a3-22c4-4cea-9b61-07fd6c842b0e","Type":"ContainerDied","Data":"6d76130398e685d08c38aedecc43b47071f6fdf36c92fd1056294f91753180b1"}
Sep 08 14:12:52.950117 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:52.950054    2561 scope.go:117] "RemoveContainer" containerID="62f86daa82ea6cdd5f99727ab46779d2ecb793ea1fee275cfd6e53284259a80e"
Sep 08 14:12:52.959172 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:52.959154    2561 scope.go:117] "RemoveContainer" containerID="62f86daa82ea6cdd5f99727ab46779d2ecb793ea1fee275cfd6e53284259a80e"
Sep 08 14:12:52.959476 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:12:52.959455    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f86daa82ea6cdd5f99727ab46779d2ecb793ea1fee275cfd6e53284259a80e\": container with ID starting with 62f86daa82ea6cdd5f99727ab46779d2ecb793ea1fee275cfd6e53284259a80e not found: ID does not exist" containerID="62f86daa82ea6cdd5f99727ab46779d2ecb793ea1fee275cfd6e53284259a80e"
Sep 08 14:12:52.959541 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:52.959484    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f86daa82ea6cdd5f99727ab46779d2ecb793ea1fee275cfd6e53284259a80e"} err="failed to get container status \"62f86daa82ea6cdd5f99727ab46779d2ecb793ea1fee275cfd6e53284259a80e\": rpc error: code = NotFound desc = could not find container \"62f86daa82ea6cdd5f99727ab46779d2ecb793ea1fee275cfd6e53284259a80e\": container with ID starting with 62f86daa82ea6cdd5f99727ab46779d2ecb793ea1fee275cfd6e53284259a80e not found: ID does not exist"
Sep 08 14:12:52.980429 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:52.980393    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb"]
Sep 08 14:12:52.982726 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:52.982698    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pr2wb"]
Sep 08 14:12:53.007317 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:12:53.007269    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ea432a3-22c4-4cea-9b61-07fd6c842b0e" path="/var/lib/kubelet/pods/8ea432a3-22c4-4cea-9b61-07fd6c842b0e/volumes"
Sep 08 14:13:19.929643 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:13:19.929607    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm"
Sep 08 14:15:12.269635 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:12.269594    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm"]
Sep 08 14:15:19.929568 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:19.929522    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm" podUID="ef354a2c-58ea-41e8-b76d-15a46e8f8999" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:15:27.291121 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:27.291058    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm" podUID="ef354a2c-58ea-41e8-b76d-15a46e8f8999" containerName="main" containerID="cri-o://39ccd7262ae55343175f0e8689e0de4dbc6400993f5a136bf6a47abdfe4f05af" gracePeriod=45
Sep 08 14:15:29.930003 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:29.929961    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm" podUID="ef354a2c-58ea-41e8-b76d-15a46e8f8999" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:15:39.929576 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:39.929538    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm" podUID="ef354a2c-58ea-41e8-b76d-15a46e8f8999" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:15:39.930096 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:39.929679    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm"
Sep 08 14:15:49.930145 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:49.930093    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm" podUID="ef354a2c-58ea-41e8-b76d-15a46e8f8999" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:15:57.588695 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:57.588617    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm"
Sep 08 14:15:57.726929 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:57.726865    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef354a2c-58ea-41e8-b76d-15a46e8f8999-tls-certs\") pod \"ef354a2c-58ea-41e8-b76d-15a46e8f8999\" (UID: \"ef354a2c-58ea-41e8-b76d-15a46e8f8999\") "
Sep 08 14:15:57.726929 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:57.726928    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbjns\" (UniqueName: \"kubernetes.io/projected/ef354a2c-58ea-41e8-b76d-15a46e8f8999-kube-api-access-kbjns\") pod \"ef354a2c-58ea-41e8-b76d-15a46e8f8999\" (UID: \"ef354a2c-58ea-41e8-b76d-15a46e8f8999\") "
Sep 08 14:15:57.729201 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:57.729172    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef354a2c-58ea-41e8-b76d-15a46e8f8999-kube-api-access-kbjns" (OuterVolumeSpecName: "kube-api-access-kbjns") pod "ef354a2c-58ea-41e8-b76d-15a46e8f8999" (UID: "ef354a2c-58ea-41e8-b76d-15a46e8f8999"). InnerVolumeSpecName "kube-api-access-kbjns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:15:57.729377 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:57.729356    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef354a2c-58ea-41e8-b76d-15a46e8f8999-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef354a2c-58ea-41e8-b76d-15a46e8f8999" (UID: "ef354a2c-58ea-41e8-b76d-15a46e8f8999"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:15:57.828006 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:57.827910    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef354a2c-58ea-41e8-b76d-15a46e8f8999-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:15:57.828006 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:57.827944    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbjns\" (UniqueName: \"kubernetes.io/projected/ef354a2c-58ea-41e8-b76d-15a46e8f8999-kube-api-access-kbjns\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:15:57.971217 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:57.971176    2561 generic.go:356] "Generic (PLEG): container finished" podID="ef354a2c-58ea-41e8-b76d-15a46e8f8999" containerID="39ccd7262ae55343175f0e8689e0de4dbc6400993f5a136bf6a47abdfe4f05af" exitCode=0
Sep 08 14:15:57.971431 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:57.971356    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm" event={"ID":"ef354a2c-58ea-41e8-b76d-15a46e8f8999","Type":"ContainerDied","Data":"39ccd7262ae55343175f0e8689e0de4dbc6400993f5a136bf6a47abdfe4f05af"}
Sep 08 14:15:57.971431 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:57.971411    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm" event={"ID":"ef354a2c-58ea-41e8-b76d-15a46e8f8999","Type":"ContainerDied","Data":"471cce2d7520961b72e708340f9a6990352779d8edefef6bddb2ee54cdde562d"}
Sep 08 14:15:57.971564 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:57.971438    2561 scope.go:117] "RemoveContainer" containerID="39ccd7262ae55343175f0e8689e0de4dbc6400993f5a136bf6a47abdfe4f05af"
Sep 08 14:15:57.971564 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:57.971491    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm"
Sep 08 14:15:57.981926 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:57.981904    2561 scope.go:117] "RemoveContainer" containerID="39ccd7262ae55343175f0e8689e0de4dbc6400993f5a136bf6a47abdfe4f05af"
Sep 08 14:15:57.982233 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:15:57.982213    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39ccd7262ae55343175f0e8689e0de4dbc6400993f5a136bf6a47abdfe4f05af\": container with ID starting with 39ccd7262ae55343175f0e8689e0de4dbc6400993f5a136bf6a47abdfe4f05af not found: ID does not exist" containerID="39ccd7262ae55343175f0e8689e0de4dbc6400993f5a136bf6a47abdfe4f05af"
Sep 08 14:15:57.982282 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:57.982242    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ccd7262ae55343175f0e8689e0de4dbc6400993f5a136bf6a47abdfe4f05af"} err="failed to get container status \"39ccd7262ae55343175f0e8689e0de4dbc6400993f5a136bf6a47abdfe4f05af\": rpc error: code = NotFound desc = could not find container \"39ccd7262ae55343175f0e8689e0de4dbc6400993f5a136bf6a47abdfe4f05af\": container with ID starting with 39ccd7262ae55343175f0e8689e0de4dbc6400993f5a136bf6a47abdfe4f05af not found: ID does not exist"
Sep 08 14:15:57.998160 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:57.998127    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm"]
Sep 08 14:15:58.006699 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:58.006673    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-lzjqm"]
Sep 08 14:15:59.008303 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:15:59.008267    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef354a2c-58ea-41e8-b76d-15a46e8f8999" path="/var/lib/kubelet/pods/ef354a2c-58ea-41e8-b76d-15a46e8f8999/volumes"
Sep 08 14:16:07.369547 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:07.369514    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"]
Sep 08 14:16:07.369956 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:07.369738    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv" podUID="d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" containerName="istio-proxy" containerID="cri-o://aff28679859364143d044153464b356acd3298cada86c69463403cc8f51774b8" gracePeriod=30
Sep 08 14:16:09.324608 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:09.324577    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 14:16:09.332706 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:09.332679    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 14:16:09.543851 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:09.543817    2561 scope.go:117] "RemoveContainer" containerID="f019cdf0526ff249116e56ac3048dcfcce3115d3e2491c4f25fcce70e6062f4c"
Sep 08 14:16:12.624364 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.624331    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:16:12.662937 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.662846    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istiod-ca-cert\") pod \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") "
Sep 08 14:16:12.662937 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.662903    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-data\") pod \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") "
Sep 08 14:16:12.662937 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.662932    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-token\") pod \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") "
Sep 08 14:16:12.665883 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.662958    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-workload-socket\") pod \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") "
Sep 08 14:16:12.665883 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.663020    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-envoy\") pod \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") "
Sep 08 14:16:12.665883 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.663080    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmx6v\" (UniqueName: \"kubernetes.io/projected/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-kube-api-access-cmx6v\") pod \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") "
Sep 08 14:16:12.665883 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.663140    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-credential-socket\") pod \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") "
Sep 08 14:16:12.665883 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.663176    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-workload-certs\") pod \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") "
Sep 08 14:16:12.665883 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.663194    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-data" (OuterVolumeSpecName: "istio-data") pod "d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" (UID: "d1b39f0f-d17f-4cdb-a61a-44c628dd7d23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:16:12.665883 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.663260    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" (UID: "d1b39f0f-d17f-4cdb-a61a-44c628dd7d23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:16:12.665883 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.663282    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-podinfo\") pod \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\" (UID: \"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23\") "
Sep 08 14:16:12.665883 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.663414    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" (UID: "d1b39f0f-d17f-4cdb-a61a-44c628dd7d23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:16:12.665883 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.663580    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istiod-ca-cert\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:16:12.665883 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.663597    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-data\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:16:12.665883 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.663610    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-workload-socket\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:16:12.665883 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.663794    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" (UID: "d1b39f0f-d17f-4cdb-a61a-44c628dd7d23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:16:12.665883 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.663867    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" (UID: "d1b39f0f-d17f-4cdb-a61a-44c628dd7d23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:16:12.666824 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.666795    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-token" (OuterVolumeSpecName: "istio-token") pod "d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" (UID: "d1b39f0f-d17f-4cdb-a61a-44c628dd7d23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:16:12.667147 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.667109    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-kube-api-access-cmx6v" (OuterVolumeSpecName: "kube-api-access-cmx6v") pod "d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" (UID: "d1b39f0f-d17f-4cdb-a61a-44c628dd7d23"). InnerVolumeSpecName "kube-api-access-cmx6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:16:12.667307 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.667265    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" (UID: "d1b39f0f-d17f-4cdb-a61a-44c628dd7d23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:16:12.667395 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.667316    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" (UID: "d1b39f0f-d17f-4cdb-a61a-44c628dd7d23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 14:16:12.764871 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.764835    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-podinfo\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:16:12.764871 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.764866    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-token\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:16:12.764871 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.764876    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-istio-envoy\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:16:12.765111 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.764885    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmx6v\" (UniqueName: \"kubernetes.io/projected/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-kube-api-access-cmx6v\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:16:12.765111 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.764895    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-credential-socket\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:16:12.765111 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:12.764903    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23-workload-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:16:13.061757 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:13.061723    2561 generic.go:356] "Generic (PLEG): container finished" podID="d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" containerID="aff28679859364143d044153464b356acd3298cada86c69463403cc8f51774b8" exitCode=0
Sep 08 14:16:13.061947 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:13.061767    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"
Sep 08 14:16:13.061947 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:13.061769    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv" event={"ID":"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23","Type":"ContainerDied","Data":"aff28679859364143d044153464b356acd3298cada86c69463403cc8f51774b8"}
Sep 08 14:16:13.061947 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:13.061903    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv" event={"ID":"d1b39f0f-d17f-4cdb-a61a-44c628dd7d23","Type":"ContainerDied","Data":"853ee11f7857d7f0675ec1a9492f8e0c47f726a8717e68b40445236cadfa312d"}
Sep 08 14:16:13.061947 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:13.061943    2561 scope.go:117] "RemoveContainer" containerID="aff28679859364143d044153464b356acd3298cada86c69463403cc8f51774b8"
Sep 08 14:16:13.071948 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:13.071930    2561 scope.go:117] "RemoveContainer" containerID="9661a365253e11b9269e67bb789b7cdcf14a8fd33628b3051010a394e6c6f1ab"
Sep 08 14:16:13.081388 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:13.081366    2561 scope.go:117] "RemoveContainer" containerID="aff28679859364143d044153464b356acd3298cada86c69463403cc8f51774b8"
Sep 08 14:16:13.081733 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:16:13.081709    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff28679859364143d044153464b356acd3298cada86c69463403cc8f51774b8\": container with ID starting with aff28679859364143d044153464b356acd3298cada86c69463403cc8f51774b8 not found: ID does not exist" containerID="aff28679859364143d044153464b356acd3298cada86c69463403cc8f51774b8"
Sep 08 14:16:13.081796 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:13.081742    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff28679859364143d044153464b356acd3298cada86c69463403cc8f51774b8"} err="failed to get container status \"aff28679859364143d044153464b356acd3298cada86c69463403cc8f51774b8\": rpc error: code = NotFound desc = could not find container \"aff28679859364143d044153464b356acd3298cada86c69463403cc8f51774b8\": container with ID starting with aff28679859364143d044153464b356acd3298cada86c69463403cc8f51774b8 not found: ID does not exist"
Sep 08 14:16:13.081796 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:13.081760    2561 scope.go:117] "RemoveContainer" containerID="9661a365253e11b9269e67bb789b7cdcf14a8fd33628b3051010a394e6c6f1ab"
Sep 08 14:16:13.081974 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:13.081953    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"]
Sep 08 14:16:13.082016 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:16:13.081997    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9661a365253e11b9269e67bb789b7cdcf14a8fd33628b3051010a394e6c6f1ab\": container with ID starting with 9661a365253e11b9269e67bb789b7cdcf14a8fd33628b3051010a394e6c6f1ab not found: ID does not exist" containerID="9661a365253e11b9269e67bb789b7cdcf14a8fd33628b3051010a394e6c6f1ab"
Sep 08 14:16:13.082051 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:13.082022    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9661a365253e11b9269e67bb789b7cdcf14a8fd33628b3051010a394e6c6f1ab"} err="failed to get container status \"9661a365253e11b9269e67bb789b7cdcf14a8fd33628b3051010a394e6c6f1ab\": rpc error: code = NotFound desc = could not find container \"9661a365253e11b9269e67bb789b7cdcf14a8fd33628b3051010a394e6c6f1ab\": container with ID starting with 9661a365253e11b9269e67bb789b7cdcf14a8fd33628b3051010a394e6c6f1ab not found: ID does not exist"
Sep 08 14:16:13.087403 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:13.087374    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-2c9jv"]
Sep 08 14:16:15.006653 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:15.006618    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" path="/var/lib/kubelet/pods/d1b39f0f-d17f-4cdb-a61a-44c628dd7d23/volumes"
Sep 08 14:16:35.106370 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.106332    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m"]
Sep 08 14:16:35.106787 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.106707    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" containerName="istio-proxy"
Sep 08 14:16:35.106787 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.106720    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" containerName="istio-proxy"
Sep 08 14:16:35.106787 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.106741    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef354a2c-58ea-41e8-b76d-15a46e8f8999" containerName="main"
Sep 08 14:16:35.106787 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.106750    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef354a2c-58ea-41e8-b76d-15a46e8f8999" containerName="main"
Sep 08 14:16:35.106787 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.106763    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ea432a3-22c4-4cea-9b61-07fd6c842b0e" containerName="main"
Sep 08 14:16:35.106787 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.106769    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ea432a3-22c4-4cea-9b61-07fd6c842b0e" containerName="main"
Sep 08 14:16:35.106979 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.106829    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" containerName="istio-proxy"
Sep 08 14:16:35.106979 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.106840    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ea432a3-22c4-4cea-9b61-07fd6c842b0e" containerName="main"
Sep 08 14:16:35.106979 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.106846    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef354a2c-58ea-41e8-b76d-15a46e8f8999" containerName="main"
Sep 08 14:16:35.109941 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.109920    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m"
Sep 08 14:16:35.114797 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.114771    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 08 14:16:35.115892 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.115864    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 08 14:16:35.116022 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.115915    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 08 14:16:35.116263 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.116241    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-tc8rj\""
Sep 08 14:16:35.117884 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.117860    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-d4mnv\""
Sep 08 14:16:35.127538 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.127494    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m"]
Sep 08 14:16:35.166348 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.166308    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9bd2740-65af-466d-a5d2-6f3425ab76f6-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m\" (UID: \"e9bd2740-65af-466d-a5d2-6f3425ab76f6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m"
Sep 08 14:16:35.166565 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.166429    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97lgt\" (UniqueName: \"kubernetes.io/projected/e9bd2740-65af-466d-a5d2-6f3425ab76f6-kube-api-access-97lgt\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m\" (UID: \"e9bd2740-65af-466d-a5d2-6f3425ab76f6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m"
Sep 08 14:16:35.267757 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.267708    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9bd2740-65af-466d-a5d2-6f3425ab76f6-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m\" (UID: \"e9bd2740-65af-466d-a5d2-6f3425ab76f6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m"
Sep 08 14:16:35.267979 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.267811    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97lgt\" (UniqueName: \"kubernetes.io/projected/e9bd2740-65af-466d-a5d2-6f3425ab76f6-kube-api-access-97lgt\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m\" (UID: \"e9bd2740-65af-466d-a5d2-6f3425ab76f6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m"
Sep 08 14:16:35.270470 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.270442    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9bd2740-65af-466d-a5d2-6f3425ab76f6-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m\" (UID: \"e9bd2740-65af-466d-a5d2-6f3425ab76f6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m"
Sep 08 14:16:35.278209 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.278179    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97lgt\" (UniqueName: \"kubernetes.io/projected/e9bd2740-65af-466d-a5d2-6f3425ab76f6-kube-api-access-97lgt\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m\" (UID: \"e9bd2740-65af-466d-a5d2-6f3425ab76f6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m"
Sep 08 14:16:35.421622 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.421511    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m"
Sep 08 14:16:35.701503 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:35.701474    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m"]
Sep 08 14:16:35.704578 ip-10-0-130-178 kubenswrapper[2561]: W0908 14:16:35.704546    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9bd2740_65af_466d_a5d2_6f3425ab76f6.slice/crio-b7e8283989aee036ea2b670871db82bfd01ce00c937e2213d9efce36237afd06 WatchSource:0}: Error finding container b7e8283989aee036ea2b670871db82bfd01ce00c937e2213d9efce36237afd06: Status 404 returned error can't find the container with id b7e8283989aee036ea2b670871db82bfd01ce00c937e2213d9efce36237afd06
Sep 08 14:16:36.191166 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:36.191128    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m" event={"ID":"e9bd2740-65af-466d-a5d2-6f3425ab76f6","Type":"ContainerStarted","Data":"55e8ff22b720991f3e71374a4e3d211637803bebc50fb418e5a921a027dd4b46"}
Sep 08 14:16:36.191707 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:36.191202    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m"
Sep 08 14:16:36.191707 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:36.191226    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m" event={"ID":"e9bd2740-65af-466d-a5d2-6f3425ab76f6","Type":"ContainerStarted","Data":"b7e8283989aee036ea2b670871db82bfd01ce00c937e2213d9efce36237afd06"}
Sep 08 14:16:36.218354 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:16:36.218281    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m" podStartSLOduration=1.218264167 podStartE2EDuration="1.218264167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:16:36.214817896 +0000 UTC m=+3627.675462707" watchObservedRunningTime="2026-09-08 14:16:36.218264167 +0000 UTC m=+3627.678908977"
Sep 08 14:17:07.199892 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:17:07.199855    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m"
Sep 08 14:18:54.373209 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:18:54.373170    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m"]
Sep 08 14:18:57.199371 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:18:57.199330    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m" podUID="e9bd2740-65af-466d-a5d2-6f3425ab76f6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:19:07.199707 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:07.199619    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m" podUID="e9bd2740-65af-466d-a5d2-6f3425ab76f6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:19:09.394654 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:09.394605    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m" podUID="e9bd2740-65af-466d-a5d2-6f3425ab76f6" containerName="main" containerID="cri-o://55e8ff22b720991f3e71374a4e3d211637803bebc50fb418e5a921a027dd4b46" gracePeriod=45
Sep 08 14:19:17.199830 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:17.199788    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m" podUID="e9bd2740-65af-466d-a5d2-6f3425ab76f6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:19:17.200224 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:17.199915    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m"
Sep 08 14:19:27.199780 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:27.199734    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m" podUID="e9bd2740-65af-466d-a5d2-6f3425ab76f6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:19:37.199477 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:37.199437    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m" podUID="e9bd2740-65af-466d-a5d2-6f3425ab76f6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:19:39.635341 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:39.635312    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m"
Sep 08 14:19:39.717929 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:39.717890    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97lgt\" (UniqueName: \"kubernetes.io/projected/e9bd2740-65af-466d-a5d2-6f3425ab76f6-kube-api-access-97lgt\") pod \"e9bd2740-65af-466d-a5d2-6f3425ab76f6\" (UID: \"e9bd2740-65af-466d-a5d2-6f3425ab76f6\") "
Sep 08 14:19:39.718108 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:39.717968    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9bd2740-65af-466d-a5d2-6f3425ab76f6-tls-certs\") pod \"e9bd2740-65af-466d-a5d2-6f3425ab76f6\" (UID: \"e9bd2740-65af-466d-a5d2-6f3425ab76f6\") "
Sep 08 14:19:39.720267 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:39.720231    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9bd2740-65af-466d-a5d2-6f3425ab76f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9bd2740-65af-466d-a5d2-6f3425ab76f6" (UID: "e9bd2740-65af-466d-a5d2-6f3425ab76f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:19:39.720413 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:39.720322    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9bd2740-65af-466d-a5d2-6f3425ab76f6-kube-api-access-97lgt" (OuterVolumeSpecName: "kube-api-access-97lgt") pod "e9bd2740-65af-466d-a5d2-6f3425ab76f6" (UID: "e9bd2740-65af-466d-a5d2-6f3425ab76f6"). InnerVolumeSpecName "kube-api-access-97lgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:19:39.819206 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:39.819166    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9bd2740-65af-466d-a5d2-6f3425ab76f6-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:19:39.819206 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:39.819199    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97lgt\" (UniqueName: \"kubernetes.io/projected/e9bd2740-65af-466d-a5d2-6f3425ab76f6-kube-api-access-97lgt\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:19:40.235233 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:40.235200    2561 generic.go:356] "Generic (PLEG): container finished" podID="e9bd2740-65af-466d-a5d2-6f3425ab76f6" containerID="55e8ff22b720991f3e71374a4e3d211637803bebc50fb418e5a921a027dd4b46" exitCode=0
Sep 08 14:19:40.235418 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:40.235242    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m"
Sep 08 14:19:40.235418 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:40.235284    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m" event={"ID":"e9bd2740-65af-466d-a5d2-6f3425ab76f6","Type":"ContainerDied","Data":"55e8ff22b720991f3e71374a4e3d211637803bebc50fb418e5a921a027dd4b46"}
Sep 08 14:19:40.235418 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:40.235360    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m" event={"ID":"e9bd2740-65af-466d-a5d2-6f3425ab76f6","Type":"ContainerDied","Data":"b7e8283989aee036ea2b670871db82bfd01ce00c937e2213d9efce36237afd06"}
Sep 08 14:19:40.235418 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:40.235383    2561 scope.go:117] "RemoveContainer" containerID="55e8ff22b720991f3e71374a4e3d211637803bebc50fb418e5a921a027dd4b46"
Sep 08 14:19:40.244744 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:40.244722    2561 scope.go:117] "RemoveContainer" containerID="55e8ff22b720991f3e71374a4e3d211637803bebc50fb418e5a921a027dd4b46"
Sep 08 14:19:40.245009 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:19:40.244989    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55e8ff22b720991f3e71374a4e3d211637803bebc50fb418e5a921a027dd4b46\": container with ID starting with 55e8ff22b720991f3e71374a4e3d211637803bebc50fb418e5a921a027dd4b46 not found: ID does not exist" containerID="55e8ff22b720991f3e71374a4e3d211637803bebc50fb418e5a921a027dd4b46"
Sep 08 14:19:40.245050 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:40.245016    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55e8ff22b720991f3e71374a4e3d211637803bebc50fb418e5a921a027dd4b46"} err="failed to get container status \"55e8ff22b720991f3e71374a4e3d211637803bebc50fb418e5a921a027dd4b46\": rpc error: code = NotFound desc = could not find container \"55e8ff22b720991f3e71374a4e3d211637803bebc50fb418e5a921a027dd4b46\": container with ID starting with 55e8ff22b720991f3e71374a4e3d211637803bebc50fb418e5a921a027dd4b46 not found: ID does not exist"
Sep 08 14:19:40.256565 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:40.256529    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m"]
Sep 08 14:19:40.260339 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:40.260311    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sched978m"]
Sep 08 14:19:41.011626 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:19:41.011592    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9bd2740-65af-466d-a5d2-6f3425ab76f6" path="/var/lib/kubelet/pods/e9bd2740-65af-466d-a5d2-6f3425ab76f6/volumes"
Sep 08 14:20:12.577932 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:12.577894    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv"]
Sep 08 14:20:12.578586 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:12.578282    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" containerName="istio-proxy"
Sep 08 14:20:12.578586 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:12.578312    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" containerName="istio-proxy"
Sep 08 14:20:12.578586 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:12.578331    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9bd2740-65af-466d-a5d2-6f3425ab76f6" containerName="main"
Sep 08 14:20:12.578586 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:12.578337    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9bd2740-65af-466d-a5d2-6f3425ab76f6" containerName="main"
Sep 08 14:20:12.578586 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:12.578399    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1b39f0f-d17f-4cdb-a61a-44c628dd7d23" containerName="istio-proxy"
Sep 08 14:20:12.578586 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:12.578413    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9bd2740-65af-466d-a5d2-6f3425ab76f6" containerName="main"
Sep 08 14:20:12.582702 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:12.582678    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv"
Sep 08 14:20:12.586913 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:12.586890    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 08 14:20:12.587082 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:12.586890    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 08 14:20:12.587728 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:12.587697    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-svv8z\""
Sep 08 14:20:12.587837 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:12.587756    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 08 14:20:12.588029 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:12.588013    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-rjpx4\""
Sep 08 14:20:12.595622 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:12.595596    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv"]
Sep 08 14:20:12.722632 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:12.722587    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5qnh\" (UniqueName: \"kubernetes.io/projected/6a7a1c09-104a-405e-a46d-7dd2c297f1bb-kube-api-access-f5qnh\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv\" (UID: \"6a7a1c09-104a-405e-a46d-7dd2c297f1bb\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv"
Sep 08 14:20:12.722824 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:12.722650    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7a1c09-104a-405e-a46d-7dd2c297f1bb-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv\" (UID: \"6a7a1c09-104a-405e-a46d-7dd2c297f1bb\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv"
Sep 08 14:20:12.823987 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:12.823946    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5qnh\" (UniqueName: \"kubernetes.io/projected/6a7a1c09-104a-405e-a46d-7dd2c297f1bb-kube-api-access-f5qnh\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv\" (UID: \"6a7a1c09-104a-405e-a46d-7dd2c297f1bb\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv"
Sep 08 14:20:12.824161 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:12.824007    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7a1c09-104a-405e-a46d-7dd2c297f1bb-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv\" (UID: \"6a7a1c09-104a-405e-a46d-7dd2c297f1bb\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv"
Sep 08 14:20:12.826640 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:12.826613    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7a1c09-104a-405e-a46d-7dd2c297f1bb-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv\" (UID: \"6a7a1c09-104a-405e-a46d-7dd2c297f1bb\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv"
Sep 08 14:20:12.833201 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:12.833141    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5qnh\" (UniqueName: \"kubernetes.io/projected/6a7a1c09-104a-405e-a46d-7dd2c297f1bb-kube-api-access-f5qnh\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv\" (UID: \"6a7a1c09-104a-405e-a46d-7dd2c297f1bb\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv"
Sep 08 14:20:12.894356 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:12.894313    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv"
Sep 08 14:20:13.170257 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:13.170223    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv"]
Sep 08 14:20:13.172818 ip-10-0-130-178 kubenswrapper[2561]: W0908 14:20:13.172791    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7a1c09_104a_405e_a46d_7dd2c297f1bb.slice/crio-4447eea849e33873716bba3ffa5cfd4052d66f7e7f6f7403e98c4dee8ff745b9 WatchSource:0}: Error finding container 4447eea849e33873716bba3ffa5cfd4052d66f7e7f6f7403e98c4dee8ff745b9: Status 404 returned error can't find the container with id 4447eea849e33873716bba3ffa5cfd4052d66f7e7f6f7403e98c4dee8ff745b9
Sep 08 14:20:13.413150 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:13.413113    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv" event={"ID":"6a7a1c09-104a-405e-a46d-7dd2c297f1bb","Type":"ContainerStarted","Data":"6be97a84581448ad191ee2b2b90b99fa7b395f8a1b044a318a16d3bd09517b69"}
Sep 08 14:20:13.413381 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:13.413176    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv" event={"ID":"6a7a1c09-104a-405e-a46d-7dd2c297f1bb","Type":"ContainerStarted","Data":"4447eea849e33873716bba3ffa5cfd4052d66f7e7f6f7403e98c4dee8ff745b9"}
Sep 08 14:20:13.413381 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:13.413221    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv"
Sep 08 14:20:13.434731 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:13.434674    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv" podStartSLOduration=1.43465686 podStartE2EDuration="1.43465686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:20:13.431464038 +0000 UTC m=+3844.892108849" watchObservedRunningTime="2026-09-08 14:20:13.43465686 +0000 UTC m=+3844.895301671"
Sep 08 14:20:44.421103 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:20:44.421070    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv"
Sep 08 14:21:09.360838 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:09.360807    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 14:21:09.373448 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:09.373418    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 14:21:20.524128 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:20.524088    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr"]
Sep 08 14:21:20.528170 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:20.528146    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr"
Sep 08 14:21:20.530477 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:20.530456    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 08 14:21:20.531254 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:20.531230    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-c46ml\""
Sep 08 14:21:20.531535 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:20.531505    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-tx2tc\""
Sep 08 14:21:20.531535 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:20.531503    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 08 14:21:20.531713 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:20.531578    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 08 14:21:20.538264 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:20.538240    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr"]
Sep 08 14:21:20.648434 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:20.648399    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2sts\" (UniqueName: \"kubernetes.io/projected/ab4d890c-72f6-4040-a4d0-b60ca1f88849-kube-api-access-b2sts\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr\" (UID: \"ab4d890c-72f6-4040-a4d0-b60ca1f88849\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr"
Sep 08 14:21:20.648609 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:20.648469    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4d890c-72f6-4040-a4d0-b60ca1f88849-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr\" (UID: \"ab4d890c-72f6-4040-a4d0-b60ca1f88849\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr"
Sep 08 14:21:20.749094 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:20.749053    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4d890c-72f6-4040-a4d0-b60ca1f88849-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr\" (UID: \"ab4d890c-72f6-4040-a4d0-b60ca1f88849\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr"
Sep 08 14:21:20.749304 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:20.749220    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2sts\" (UniqueName: \"kubernetes.io/projected/ab4d890c-72f6-4040-a4d0-b60ca1f88849-kube-api-access-b2sts\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr\" (UID: \"ab4d890c-72f6-4040-a4d0-b60ca1f88849\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr"
Sep 08 14:21:20.751846 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:20.751818    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4d890c-72f6-4040-a4d0-b60ca1f88849-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr\" (UID: \"ab4d890c-72f6-4040-a4d0-b60ca1f88849\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr"
Sep 08 14:21:20.758273 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:20.758239    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2sts\" (UniqueName: \"kubernetes.io/projected/ab4d890c-72f6-4040-a4d0-b60ca1f88849-kube-api-access-b2sts\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr\" (UID: \"ab4d890c-72f6-4040-a4d0-b60ca1f88849\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr"
Sep 08 14:21:20.839201 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:20.839102    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr"
Sep 08 14:21:21.127474 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:21.127448    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr"]
Sep 08 14:21:21.130085 ip-10-0-130-178 kubenswrapper[2561]: W0908 14:21:21.130048    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4d890c_72f6_4040_a4d0_b60ca1f88849.slice/crio-d16f68b4599abee2390d6fe2c1882cfda2df6fc042dab39876265fcb08745469 WatchSource:0}: Error finding container d16f68b4599abee2390d6fe2c1882cfda2df6fc042dab39876265fcb08745469: Status 404 returned error can't find the container with id d16f68b4599abee2390d6fe2c1882cfda2df6fc042dab39876265fcb08745469
Sep 08 14:21:21.798216 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:21.798168    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr" event={"ID":"ab4d890c-72f6-4040-a4d0-b60ca1f88849","Type":"ContainerStarted","Data":"139af6d71259c6ee4282316e86b2b0e77d1a9d7064ef6d3694304e7b88ac4093"}
Sep 08 14:21:21.798625 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:21.798269    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr" event={"ID":"ab4d890c-72f6-4040-a4d0-b60ca1f88849","Type":"ContainerStarted","Data":"d16f68b4599abee2390d6fe2c1882cfda2df6fc042dab39876265fcb08745469"}
Sep 08 14:21:21.798625 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:21.798326    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr"
Sep 08 14:21:21.819692 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:21.819638    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr" podStartSLOduration=1.819623443 podStartE2EDuration="1.819623443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:21:21.817062839 +0000 UTC m=+3913.277707664" watchObservedRunningTime="2026-09-08 14:21:21.819623443 +0000 UTC m=+3913.280268252"
Sep 08 14:21:52.808859 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:21:52.808821    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr"
Sep 08 14:22:21.677805 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:22:21.677725    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv"]
Sep 08 14:22:24.420684 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:22:24.420647    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv" podUID="6a7a1c09-104a-405e-a46d-7dd2c297f1bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:22:34.420709 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:22:34.420665    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv" podUID="6a7a1c09-104a-405e-a46d-7dd2c297f1bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:22:36.699508 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:22:36.699463    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv" podUID="6a7a1c09-104a-405e-a46d-7dd2c297f1bb" containerName="main" containerID="cri-o://6be97a84581448ad191ee2b2b90b99fa7b395f8a1b044a318a16d3bd09517b69" gracePeriod=45
Sep 08 14:22:44.420728 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:22:44.420689    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv" podUID="6a7a1c09-104a-405e-a46d-7dd2c297f1bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:22:44.421122 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:22:44.420792    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv"
Sep 08 14:22:54.420775 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:22:54.420737    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv" podUID="6a7a1c09-104a-405e-a46d-7dd2c297f1bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:23:04.420354 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:04.420312    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv" podUID="6a7a1c09-104a-405e-a46d-7dd2c297f1bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:23:06.953757 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:06.953733    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv"
Sep 08 14:23:07.078777 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:07.078672    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5qnh\" (UniqueName: \"kubernetes.io/projected/6a7a1c09-104a-405e-a46d-7dd2c297f1bb-kube-api-access-f5qnh\") pod \"6a7a1c09-104a-405e-a46d-7dd2c297f1bb\" (UID: \"6a7a1c09-104a-405e-a46d-7dd2c297f1bb\") "
Sep 08 14:23:07.078931 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:07.078823    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7a1c09-104a-405e-a46d-7dd2c297f1bb-tls-certs\") pod \"6a7a1c09-104a-405e-a46d-7dd2c297f1bb\" (UID: \"6a7a1c09-104a-405e-a46d-7dd2c297f1bb\") "
Sep 08 14:23:07.081204 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:07.081172    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7a1c09-104a-405e-a46d-7dd2c297f1bb-kube-api-access-f5qnh" (OuterVolumeSpecName: "kube-api-access-f5qnh") pod "6a7a1c09-104a-405e-a46d-7dd2c297f1bb" (UID: "6a7a1c09-104a-405e-a46d-7dd2c297f1bb"). InnerVolumeSpecName "kube-api-access-f5qnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:23:07.081346 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:07.081216    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7a1c09-104a-405e-a46d-7dd2c297f1bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a7a1c09-104a-405e-a46d-7dd2c297f1bb" (UID: "6a7a1c09-104a-405e-a46d-7dd2c297f1bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:23:07.180511 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:07.180475    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7a1c09-104a-405e-a46d-7dd2c297f1bb-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:23:07.180511 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:07.180506    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5qnh\" (UniqueName: \"kubernetes.io/projected/6a7a1c09-104a-405e-a46d-7dd2c297f1bb-kube-api-access-f5qnh\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:23:07.358891 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:07.358797    2561 generic.go:356] "Generic (PLEG): container finished" podID="6a7a1c09-104a-405e-a46d-7dd2c297f1bb" containerID="6be97a84581448ad191ee2b2b90b99fa7b395f8a1b044a318a16d3bd09517b69" exitCode=0
Sep 08 14:23:07.358891 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:07.358852    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv"
Sep 08 14:23:07.359101 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:07.358885    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv" event={"ID":"6a7a1c09-104a-405e-a46d-7dd2c297f1bb","Type":"ContainerDied","Data":"6be97a84581448ad191ee2b2b90b99fa7b395f8a1b044a318a16d3bd09517b69"}
Sep 08 14:23:07.359101 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:07.358942    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv" event={"ID":"6a7a1c09-104a-405e-a46d-7dd2c297f1bb","Type":"ContainerDied","Data":"4447eea849e33873716bba3ffa5cfd4052d66f7e7f6f7403e98c4dee8ff745b9"}
Sep 08 14:23:07.359101 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:07.358965    2561 scope.go:117] "RemoveContainer" containerID="6be97a84581448ad191ee2b2b90b99fa7b395f8a1b044a318a16d3bd09517b69"
Sep 08 14:23:07.368467 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:07.368446    2561 scope.go:117] "RemoveContainer" containerID="6be97a84581448ad191ee2b2b90b99fa7b395f8a1b044a318a16d3bd09517b69"
Sep 08 14:23:07.368738 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:23:07.368721    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6be97a84581448ad191ee2b2b90b99fa7b395f8a1b044a318a16d3bd09517b69\": container with ID starting with 6be97a84581448ad191ee2b2b90b99fa7b395f8a1b044a318a16d3bd09517b69 not found: ID does not exist" containerID="6be97a84581448ad191ee2b2b90b99fa7b395f8a1b044a318a16d3bd09517b69"
Sep 08 14:23:07.368793 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:07.368746    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6be97a84581448ad191ee2b2b90b99fa7b395f8a1b044a318a16d3bd09517b69"} err="failed to get container status \"6be97a84581448ad191ee2b2b90b99fa7b395f8a1b044a318a16d3bd09517b69\": rpc error: code = NotFound desc = could not find container \"6be97a84581448ad191ee2b2b90b99fa7b395f8a1b044a318a16d3bd09517b69\": container with ID starting with 6be97a84581448ad191ee2b2b90b99fa7b395f8a1b044a318a16d3bd09517b69 not found: ID does not exist"
Sep 08 14:23:07.381958 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:07.381925    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv"]
Sep 08 14:23:07.385034 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:07.385009    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485wqcv"]
Sep 08 14:23:09.008096 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:09.008066    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a7a1c09-104a-405e-a46d-7dd2c297f1bb" path="/var/lib/kubelet/pods/6a7a1c09-104a-405e-a46d-7dd2c297f1bb/volumes"
Sep 08 14:23:12.068611 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.068569    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"]
Sep 08 14:23:12.069085 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.068952    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a7a1c09-104a-405e-a46d-7dd2c297f1bb" containerName="main"
Sep 08 14:23:12.069085 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.068967    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a7a1c09-104a-405e-a46d-7dd2c297f1bb" containerName="main"
Sep 08 14:23:12.069085 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.069059    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a7a1c09-104a-405e-a46d-7dd2c297f1bb" containerName="main"
Sep 08 14:23:12.073842 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.073811    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.095427 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.095386    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-sb5bt\""
Sep 08 14:23:12.095928 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.095900    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 08 14:23:12.096032 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.095900    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 08 14:23:12.098487 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.098463    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 08 14:23:12.103564 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.103536    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"]
Sep 08 14:23:12.125923 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.125879    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-workload-certs\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.126085 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.125929    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxbdq\" (UniqueName: \"kubernetes.io/projected/47d85102-7c7d-4aad-aa92-d9e41a7ea535-kube-api-access-wxbdq\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.126085 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.125977    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-data\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.126085 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.126009    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-credential-socket\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.126275 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.126087    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-podinfo\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.126275 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.126121    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.126275 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.126142    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-envoy\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.126275 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.126167    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-workload-socket\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.126275 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.126217    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-token\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.227614 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.227548    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-workload-certs\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.227614 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.227611    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxbdq\" (UniqueName: \"kubernetes.io/projected/47d85102-7c7d-4aad-aa92-d9e41a7ea535-kube-api-access-wxbdq\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.227900 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.227655    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-data\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.227900 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.227692    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-credential-socket\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.227900 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.227733    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-podinfo\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.227900 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.227759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.227900 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.227789    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-envoy\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.227900 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.227822    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-workload-socket\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.227900 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.227845    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-token\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.228393 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.228113    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-workload-certs\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.228644 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.228600    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-data\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.228644 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.228633    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-credential-socket\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.228841 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.228756    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-workload-socket\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.228841 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.228833    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.230512 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.230486    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-envoy\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.230774 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.230757    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-podinfo\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.250630 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.250599    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxbdq\" (UniqueName: \"kubernetes.io/projected/47d85102-7c7d-4aad-aa92-d9e41a7ea535-kube-api-access-wxbdq\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.256228 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.256186    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-token\") pod \"router-gateway-2-openshift-default-5c874f788-4vkxn\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.384175 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.384068    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:12.663909 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.663868    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"]
Sep 08 14:23:12.665732 ip-10-0-130-178 kubenswrapper[2561]: W0908 14:23:12.665710    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d85102_7c7d_4aad_aa92_d9e41a7ea535.slice/crio-e476505f7e1dd6734935cbc72c93996ba43531ee22453a0f7a1fddbb70ea7e57 WatchSource:0}: Error finding container e476505f7e1dd6734935cbc72c93996ba43531ee22453a0f7a1fddbb70ea7e57: Status 404 returned error can't find the container with id e476505f7e1dd6734935cbc72c93996ba43531ee22453a0f7a1fddbb70ea7e57
Sep 08 14:23:12.667821 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.667788    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 14:23:12.667908 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.667855    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 14:23:12.667908 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:12.667883    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 14:23:13.393254 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:13.393215    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn" event={"ID":"47d85102-7c7d-4aad-aa92-d9e41a7ea535","Type":"ContainerStarted","Data":"b8cd9244374d1b9d662108b7e32cb0f5d5520708b648b28ce60b9d706ddead17"}
Sep 08 14:23:13.393763 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:13.393270    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn" event={"ID":"47d85102-7c7d-4aad-aa92-d9e41a7ea535","Type":"ContainerStarted","Data":"e476505f7e1dd6734935cbc72c93996ba43531ee22453a0f7a1fddbb70ea7e57"}
Sep 08 14:23:13.414971 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:13.414910    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn" podStartSLOduration=1.414889824 podStartE2EDuration="1.414889824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:23:13.414689078 +0000 UTC m=+4024.875333889" watchObservedRunningTime="2026-09-08 14:23:13.414889824 +0000 UTC m=+4024.875534638"
Sep 08 14:23:14.384905 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:14.384863    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:14.395133 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:14.395099    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:14.399777 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:14.399744    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:14.400582 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:14.400563    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:24.460207 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:24.460108    2561 generic.go:356] "Generic (PLEG): container finished" podID="47d85102-7c7d-4aad-aa92-d9e41a7ea535" containerID="b8cd9244374d1b9d662108b7e32cb0f5d5520708b648b28ce60b9d706ddead17" exitCode=0
Sep 08 14:23:24.460207 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:24.460178    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn" event={"ID":"47d85102-7c7d-4aad-aa92-d9e41a7ea535","Type":"ContainerDied","Data":"b8cd9244374d1b9d662108b7e32cb0f5d5520708b648b28ce60b9d706ddead17"}
Sep 08 14:23:24.461025 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:24.460998    2561 scope.go:117] "RemoveContainer" containerID="b8cd9244374d1b9d662108b7e32cb0f5d5520708b648b28ce60b9d706ddead17"
Sep 08 14:23:24.462306 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:24.462252    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 14:23:24.462393 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:24.462348    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 14:23:24.462393 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:24.462381    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 14:23:25.471363 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:25.471326    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn" event={"ID":"47d85102-7c7d-4aad-aa92-d9e41a7ea535","Type":"ContainerStarted","Data":"c6ebb25b0db2a8776fe61137ff160bc9b24ea0610b5f423632aa2962421a837b"}
Sep 08 14:23:26.385055 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:26.385014    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:26.386405 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:26.386366    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn" podUID="47d85102-7c7d-4aad-aa92-d9e41a7ea535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Sep 08 14:23:27.385339 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:27.385270    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn" podUID="47d85102-7c7d-4aad-aa92-d9e41a7ea535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Sep 08 14:23:28.384741 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:28.384701    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn" podUID="47d85102-7c7d-4aad-aa92-d9e41a7ea535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Sep 08 14:23:29.384658 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:29.384620    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn" podUID="47d85102-7c7d-4aad-aa92-d9e41a7ea535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Sep 08 14:23:29.400708 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:29.400681    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:30.390331 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:30.390280    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:30.391545 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:30.391523    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:23:31.141914 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:31.141760    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68"]
Sep 08 14:23:31.145485 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:31.145459    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68"
Sep 08 14:23:31.148575 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:31.148552    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-m98fm\""
Sep 08 14:23:31.148732 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:31.148560    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-bs7m2\""
Sep 08 14:23:31.148808 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:31.148581    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 08 14:23:31.154170 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:31.154146    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68"]
Sep 08 14:23:31.212230 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:31.212185    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c08b1cbf-dc35-4663-aa95-1ef875ebba98-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68\" (UID: \"c08b1cbf-dc35-4663-aa95-1ef875ebba98\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68"
Sep 08 14:23:31.212230 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:31.212229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjwzq\" (UniqueName: \"kubernetes.io/projected/c08b1cbf-dc35-4663-aa95-1ef875ebba98-kube-api-access-xjwzq\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68\" (UID: \"c08b1cbf-dc35-4663-aa95-1ef875ebba98\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68"
Sep 08 14:23:31.313346 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:31.313315    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c08b1cbf-dc35-4663-aa95-1ef875ebba98-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68\" (UID: \"c08b1cbf-dc35-4663-aa95-1ef875ebba98\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68"
Sep 08 14:23:31.313346 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:31.313347    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjwzq\" (UniqueName: \"kubernetes.io/projected/c08b1cbf-dc35-4663-aa95-1ef875ebba98-kube-api-access-xjwzq\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68\" (UID: \"c08b1cbf-dc35-4663-aa95-1ef875ebba98\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68"
Sep 08 14:23:31.315965 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:31.315938    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c08b1cbf-dc35-4663-aa95-1ef875ebba98-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68\" (UID: \"c08b1cbf-dc35-4663-aa95-1ef875ebba98\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68"
Sep 08 14:23:31.322556 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:31.322531    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjwzq\" (UniqueName: \"kubernetes.io/projected/c08b1cbf-dc35-4663-aa95-1ef875ebba98-kube-api-access-xjwzq\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68\" (UID: \"c08b1cbf-dc35-4663-aa95-1ef875ebba98\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68"
Sep 08 14:23:31.458115 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:31.458071    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68"
Sep 08 14:23:31.740183 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:31.740155    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68"]
Sep 08 14:23:31.742070 ip-10-0-130-178 kubenswrapper[2561]: W0908 14:23:31.742041    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08b1cbf_dc35_4663_aa95_1ef875ebba98.slice/crio-a0679f6773a797ad619b12b24bb55ea4582970df22c1193124fa5878fbef9d20 WatchSource:0}: Error finding container a0679f6773a797ad619b12b24bb55ea4582970df22c1193124fa5878fbef9d20: Status 404 returned error can't find the container with id a0679f6773a797ad619b12b24bb55ea4582970df22c1193124fa5878fbef9d20
Sep 08 14:23:32.511842 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:32.511806    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68" event={"ID":"c08b1cbf-dc35-4663-aa95-1ef875ebba98","Type":"ContainerStarted","Data":"948d8055198747b3eb8cbd5ce08974ef674d47b6c8cdfab03c0087e3ca8aaa04"}
Sep 08 14:23:32.512267 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:32.511866    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68" event={"ID":"c08b1cbf-dc35-4663-aa95-1ef875ebba98","Type":"ContainerStarted","Data":"a0679f6773a797ad619b12b24bb55ea4582970df22c1193124fa5878fbef9d20"}
Sep 08 14:23:32.512267 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:32.512017    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68"
Sep 08 14:23:32.532976 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:32.532925    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68" podStartSLOduration=1.532907998 podStartE2EDuration="1.532907998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:23:32.530121478 +0000 UTC m=+4043.990766285" watchObservedRunningTime="2026-09-08 14:23:32.532907998 +0000 UTC m=+4043.993552809"
Sep 08 14:23:48.624172 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:48.624132    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr"]
Sep 08 14:23:52.807801 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:23:52.807758    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr" podUID="ab4d890c-72f6-4040-a4d0-b60ca1f88849" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:24:02.808795 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:02.808754    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr" podUID="ab4d890c-72f6-4040-a4d0-b60ca1f88849" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:24:03.518814 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:03.518783    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68"
Sep 08 14:24:03.646965 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:03.646917    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr" podUID="ab4d890c-72f6-4040-a4d0-b60ca1f88849" containerName="main" containerID="cri-o://139af6d71259c6ee4282316e86b2b0e77d1a9d7064ef6d3694304e7b88ac4093" gracePeriod=45
Sep 08 14:24:12.807790 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:12.807745    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr" podUID="ab4d890c-72f6-4040-a4d0-b60ca1f88849" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:24:12.808226 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:12.807869    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr"
Sep 08 14:24:22.808359 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:22.808313    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr" podUID="ab4d890c-72f6-4040-a4d0-b60ca1f88849" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:24:32.808411 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:32.808367    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr" podUID="ab4d890c-72f6-4040-a4d0-b60ca1f88849" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:24:33.855712 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:33.855671    2561 generic.go:356] "Generic (PLEG): container finished" podID="ab4d890c-72f6-4040-a4d0-b60ca1f88849" containerID="139af6d71259c6ee4282316e86b2b0e77d1a9d7064ef6d3694304e7b88ac4093" exitCode=0
Sep 08 14:24:33.856165 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:33.855740    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr" event={"ID":"ab4d890c-72f6-4040-a4d0-b60ca1f88849","Type":"ContainerDied","Data":"139af6d71259c6ee4282316e86b2b0e77d1a9d7064ef6d3694304e7b88ac4093"}
Sep 08 14:24:33.903273 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:33.903244    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr"
Sep 08 14:24:33.975157 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:33.975120    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4d890c-72f6-4040-a4d0-b60ca1f88849-tls-certs\") pod \"ab4d890c-72f6-4040-a4d0-b60ca1f88849\" (UID: \"ab4d890c-72f6-4040-a4d0-b60ca1f88849\") "
Sep 08 14:24:33.975368 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:33.975192    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2sts\" (UniqueName: \"kubernetes.io/projected/ab4d890c-72f6-4040-a4d0-b60ca1f88849-kube-api-access-b2sts\") pod \"ab4d890c-72f6-4040-a4d0-b60ca1f88849\" (UID: \"ab4d890c-72f6-4040-a4d0-b60ca1f88849\") "
Sep 08 14:24:33.977462 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:33.977430    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab4d890c-72f6-4040-a4d0-b60ca1f88849-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab4d890c-72f6-4040-a4d0-b60ca1f88849" (UID: "ab4d890c-72f6-4040-a4d0-b60ca1f88849"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:24:33.977572 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:33.977486    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4d890c-72f6-4040-a4d0-b60ca1f88849-kube-api-access-b2sts" (OuterVolumeSpecName: "kube-api-access-b2sts") pod "ab4d890c-72f6-4040-a4d0-b60ca1f88849" (UID: "ab4d890c-72f6-4040-a4d0-b60ca1f88849"). InnerVolumeSpecName "kube-api-access-b2sts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:24:34.076640 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:34.076595    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4d890c-72f6-4040-a4d0-b60ca1f88849-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:24:34.076640 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:34.076633    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2sts\" (UniqueName: \"kubernetes.io/projected/ab4d890c-72f6-4040-a4d0-b60ca1f88849-kube-api-access-b2sts\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:24:34.862417 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:34.862383    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr"
Sep 08 14:24:34.862890 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:34.862423    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr" event={"ID":"ab4d890c-72f6-4040-a4d0-b60ca1f88849","Type":"ContainerDied","Data":"d16f68b4599abee2390d6fe2c1882cfda2df6fc042dab39876265fcb08745469"}
Sep 08 14:24:34.862890 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:34.862496    2561 scope.go:117] "RemoveContainer" containerID="139af6d71259c6ee4282316e86b2b0e77d1a9d7064ef6d3694304e7b88ac4093"
Sep 08 14:24:34.885844 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:34.885817    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr"]
Sep 08 14:24:34.897698 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:34.897675    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-x4dbr"]
Sep 08 14:24:35.007641 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:24:35.007596    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab4d890c-72f6-4040-a4d0-b60ca1f88849" path="/var/lib/kubelet/pods/ab4d890c-72f6-4040-a4d0-b60ca1f88849/volumes"
Sep 08 14:25:40.376754 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:25:40.376715    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68"]
Sep 08 14:25:43.519103 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:25:43.519059    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68" podUID="c08b1cbf-dc35-4663-aa95-1ef875ebba98" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:25:53.519231 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:25:53.519185    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68" podUID="c08b1cbf-dc35-4663-aa95-1ef875ebba98" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:25:55.399124 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:25:55.399078    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68" podUID="c08b1cbf-dc35-4663-aa95-1ef875ebba98" containerName="main" containerID="cri-o://948d8055198747b3eb8cbd5ce08974ef674d47b6c8cdfab03c0087e3ca8aaa04" gracePeriod=45
Sep 08 14:26:03.519167 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:03.519123    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68" podUID="c08b1cbf-dc35-4663-aa95-1ef875ebba98" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:26:03.519581 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:03.519248    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68"
Sep 08 14:26:09.398420 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:09.398391    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 14:26:09.408899 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:09.408873    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 14:26:13.518929 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:13.518887    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68" podUID="c08b1cbf-dc35-4663-aa95-1ef875ebba98" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:26:23.518999 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:23.518950    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68" podUID="c08b1cbf-dc35-4663-aa95-1ef875ebba98" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:26:25.657659 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:25.657628    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68"
Sep 08 14:26:25.790839 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:25.790796    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c08b1cbf-dc35-4663-aa95-1ef875ebba98-tls-certs\") pod \"c08b1cbf-dc35-4663-aa95-1ef875ebba98\" (UID: \"c08b1cbf-dc35-4663-aa95-1ef875ebba98\") "
Sep 08 14:26:25.791026 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:25.790913    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjwzq\" (UniqueName: \"kubernetes.io/projected/c08b1cbf-dc35-4663-aa95-1ef875ebba98-kube-api-access-xjwzq\") pod \"c08b1cbf-dc35-4663-aa95-1ef875ebba98\" (UID: \"c08b1cbf-dc35-4663-aa95-1ef875ebba98\") "
Sep 08 14:26:25.793267 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:25.793234    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08b1cbf-dc35-4663-aa95-1ef875ebba98-kube-api-access-xjwzq" (OuterVolumeSpecName: "kube-api-access-xjwzq") pod "c08b1cbf-dc35-4663-aa95-1ef875ebba98" (UID: "c08b1cbf-dc35-4663-aa95-1ef875ebba98"). InnerVolumeSpecName "kube-api-access-xjwzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:26:25.793377 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:25.793249    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c08b1cbf-dc35-4663-aa95-1ef875ebba98-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c08b1cbf-dc35-4663-aa95-1ef875ebba98" (UID: "c08b1cbf-dc35-4663-aa95-1ef875ebba98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:26:25.891506 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:25.891467    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjwzq\" (UniqueName: \"kubernetes.io/projected/c08b1cbf-dc35-4663-aa95-1ef875ebba98-kube-api-access-xjwzq\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:26:25.891506 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:25.891500    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c08b1cbf-dc35-4663-aa95-1ef875ebba98-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:26:26.467282 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:26.467242    2561 generic.go:356] "Generic (PLEG): container finished" podID="c08b1cbf-dc35-4663-aa95-1ef875ebba98" containerID="948d8055198747b3eb8cbd5ce08974ef674d47b6c8cdfab03c0087e3ca8aaa04" exitCode=0
Sep 08 14:26:26.467495 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:26.467314    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68"
Sep 08 14:26:26.467495 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:26.467325    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68" event={"ID":"c08b1cbf-dc35-4663-aa95-1ef875ebba98","Type":"ContainerDied","Data":"948d8055198747b3eb8cbd5ce08974ef674d47b6c8cdfab03c0087e3ca8aaa04"}
Sep 08 14:26:26.467495 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:26.467380    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68" event={"ID":"c08b1cbf-dc35-4663-aa95-1ef875ebba98","Type":"ContainerDied","Data":"a0679f6773a797ad619b12b24bb55ea4582970df22c1193124fa5878fbef9d20"}
Sep 08 14:26:26.467495 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:26.467404    2561 scope.go:117] "RemoveContainer" containerID="948d8055198747b3eb8cbd5ce08974ef674d47b6c8cdfab03c0087e3ca8aaa04"
Sep 08 14:26:26.476936 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:26.476912    2561 scope.go:117] "RemoveContainer" containerID="948d8055198747b3eb8cbd5ce08974ef674d47b6c8cdfab03c0087e3ca8aaa04"
Sep 08 14:26:26.477276 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:26:26.477257    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"948d8055198747b3eb8cbd5ce08974ef674d47b6c8cdfab03c0087e3ca8aaa04\": container with ID starting with 948d8055198747b3eb8cbd5ce08974ef674d47b6c8cdfab03c0087e3ca8aaa04 not found: ID does not exist" containerID="948d8055198747b3eb8cbd5ce08974ef674d47b6c8cdfab03c0087e3ca8aaa04"
Sep 08 14:26:26.477346 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:26.477302    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"948d8055198747b3eb8cbd5ce08974ef674d47b6c8cdfab03c0087e3ca8aaa04"} err="failed to get container status \"948d8055198747b3eb8cbd5ce08974ef674d47b6c8cdfab03c0087e3ca8aaa04\": rpc error: code = NotFound desc = could not find container \"948d8055198747b3eb8cbd5ce08974ef674d47b6c8cdfab03c0087e3ca8aaa04\": container with ID starting with 948d8055198747b3eb8cbd5ce08974ef674d47b6c8cdfab03c0087e3ca8aaa04 not found: ID does not exist"
Sep 08 14:26:26.491147 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:26.490960    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68"]
Sep 08 14:26:26.493641 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:26.493614    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45l4x68"]
Sep 08 14:26:27.006801 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:27.006768    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c08b1cbf-dc35-4663-aa95-1ef875ebba98" path="/var/lib/kubelet/pods/c08b1cbf-dc35-4663-aa95-1ef875ebba98/volumes"
Sep 08 14:26:35.498737 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.498656    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"]
Sep 08 14:26:35.499092 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.499070    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c08b1cbf-dc35-4663-aa95-1ef875ebba98" containerName="main"
Sep 08 14:26:35.499092 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.499083    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c08b1cbf-dc35-4663-aa95-1ef875ebba98" containerName="main"
Sep 08 14:26:35.499171 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.499095    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab4d890c-72f6-4040-a4d0-b60ca1f88849" containerName="main"
Sep 08 14:26:35.499171 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.499101    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab4d890c-72f6-4040-a4d0-b60ca1f88849" containerName="main"
Sep 08 14:26:35.499171 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.499164    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab4d890c-72f6-4040-a4d0-b60ca1f88849" containerName="main"
Sep 08 14:26:35.499265 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.499175    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c08b1cbf-dc35-4663-aa95-1ef875ebba98" containerName="main"
Sep 08 14:26:35.502236 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.502208    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:35.504621 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.504592    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 08 14:26:35.504754 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.504597    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 08 14:26:35.505966 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.505346    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-flfhf\""
Sep 08 14:26:35.505966 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.505495    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 08 14:26:35.512696 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.512671    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"]
Sep 08 14:26:35.644231 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.644184    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"]
Sep 08 14:26:35.644491 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.644461    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn" podUID="47d85102-7c7d-4aad-aa92-d9e41a7ea535" containerName="istio-proxy" containerID="cri-o://c6ebb25b0db2a8776fe61137ff160bc9b24ea0610b5f423632aa2962421a837b" gracePeriod=30
Sep 08 14:26:35.678239 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.678203    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h6bd6\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:35.678239 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.678245    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h6bd6\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:35.678521 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.678336    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l76t8\" (UniqueName: \"kubernetes.io/projected/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-kube-api-access-l76t8\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h6bd6\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:35.678521 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.678390    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h6bd6\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:35.678521 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.678413    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h6bd6\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:35.678521 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.678459    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h6bd6\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:35.719171 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.719134    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w"]
Sep 08 14:26:35.723139 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.723111    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w"
Sep 08 14:26:35.725847 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.725824    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-qzww9\""
Sep 08 14:26:35.736104 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.736077    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w"]
Sep 08 14:26:35.779371 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.779253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l76t8\" (UniqueName: \"kubernetes.io/projected/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-kube-api-access-l76t8\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h6bd6\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:35.779371 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.779353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h6bd6\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:35.779595 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.779381    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h6bd6\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:35.779595 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.779407    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h6bd6\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:35.779595 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.779462    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h6bd6\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:35.779595 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.779483    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h6bd6\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:35.779880 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.779855    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h6bd6\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:35.779946 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.779863    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h6bd6\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:35.780002 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.779984    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h6bd6\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:35.781880 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.781851    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h6bd6\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:35.782315 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.782276    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h6bd6\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:35.793057 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.793021    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l76t8\" (UniqueName: \"kubernetes.io/projected/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-kube-api-access-l76t8\") pod \"rolling-upgrade-test-kserve-74b79d7f77-h6bd6\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:35.815957 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.815925    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:35.881835 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.881133    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsqb4\" (UniqueName: \"kubernetes.io/projected/347d5927-f0c3-46fb-8ac7-4971f7f0f786-kube-api-access-tsqb4\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w\" (UID: \"347d5927-f0c3-46fb-8ac7-4971f7f0f786\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w"
Sep 08 14:26:35.881835 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.881189    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347d5927-f0c3-46fb-8ac7-4971f7f0f786-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w\" (UID: \"347d5927-f0c3-46fb-8ac7-4971f7f0f786\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w"
Sep 08 14:26:35.982397 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.982355    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsqb4\" (UniqueName: \"kubernetes.io/projected/347d5927-f0c3-46fb-8ac7-4971f7f0f786-kube-api-access-tsqb4\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w\" (UID: \"347d5927-f0c3-46fb-8ac7-4971f7f0f786\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w"
Sep 08 14:26:35.982607 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.982405    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347d5927-f0c3-46fb-8ac7-4971f7f0f786-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w\" (UID: \"347d5927-f0c3-46fb-8ac7-4971f7f0f786\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w"
Sep 08 14:26:35.985006 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.984976    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347d5927-f0c3-46fb-8ac7-4971f7f0f786-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w\" (UID: \"347d5927-f0c3-46fb-8ac7-4971f7f0f786\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w"
Sep 08 14:26:35.990851 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:35.990824    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsqb4\" (UniqueName: \"kubernetes.io/projected/347d5927-f0c3-46fb-8ac7-4971f7f0f786-kube-api-access-tsqb4\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w\" (UID: \"347d5927-f0c3-46fb-8ac7-4971f7f0f786\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w"
Sep 08 14:26:36.034547 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:36.034439    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w"
Sep 08 14:26:36.095011 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:36.094110    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"]
Sep 08 14:26:36.096662 ip-10-0-130-178 kubenswrapper[2561]: W0908 14:26:36.096630    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb5bc8a_bf24_4bc5_bca4_9f1a9736e910.slice/crio-58a4fb61bf596934734ba8448dacf72e35fd34cce036811b900b9c584ce76eb7 WatchSource:0}: Error finding container 58a4fb61bf596934734ba8448dacf72e35fd34cce036811b900b9c584ce76eb7: Status 404 returned error can't find the container with id 58a4fb61bf596934734ba8448dacf72e35fd34cce036811b900b9c584ce76eb7
Sep 08 14:26:36.308259 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:36.308227    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w"]
Sep 08 14:26:36.310120 ip-10-0-130-178 kubenswrapper[2561]: W0908 14:26:36.310089    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347d5927_f0c3_46fb_8ac7_4971f7f0f786.slice/crio-76114e084b9b614feba03cb0cba0501cf6ef8ac006f4a1d6f37ebb1fbacc16bf WatchSource:0}: Error finding container 76114e084b9b614feba03cb0cba0501cf6ef8ac006f4a1d6f37ebb1fbacc16bf: Status 404 returned error can't find the container with id 76114e084b9b614feba03cb0cba0501cf6ef8ac006f4a1d6f37ebb1fbacc16bf
Sep 08 14:26:36.524250 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:36.524213    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6" event={"ID":"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910","Type":"ContainerStarted","Data":"043015e38712a29db94316de315731ff9d96b9d3513ee4729d0eebcd29857d08"}
Sep 08 14:26:36.524690 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:36.524273    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6" event={"ID":"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910","Type":"ContainerStarted","Data":"58a4fb61bf596934734ba8448dacf72e35fd34cce036811b900b9c584ce76eb7"}
Sep 08 14:26:36.526077 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:36.526053    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w" event={"ID":"347d5927-f0c3-46fb-8ac7-4971f7f0f786","Type":"ContainerStarted","Data":"dbaab007ef4f6ebec519d99217f1546d5b990c9b2b498cde05aaf575bf9c08fa"}
Sep 08 14:26:36.526214 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:36.526104    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w"
Sep 08 14:26:36.526214 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:36.526123    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w" event={"ID":"347d5927-f0c3-46fb-8ac7-4971f7f0f786","Type":"ContainerStarted","Data":"76114e084b9b614feba03cb0cba0501cf6ef8ac006f4a1d6f37ebb1fbacc16bf"}
Sep 08 14:26:36.547459 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:36.547405    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6" podStartSLOduration=1.547385429 podStartE2EDuration="1.547385429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:26:36.545682053 +0000 UTC m=+4228.006326863" watchObservedRunningTime="2026-09-08 14:26:36.547385429 +0000 UTC m=+4228.008030238"
Sep 08 14:26:36.568637 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:36.568519    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w" podStartSLOduration=1.568495806 podStartE2EDuration="1.568495806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:26:36.566433232 +0000 UTC m=+4228.027078043" watchObservedRunningTime="2026-09-08 14:26:36.568495806 +0000 UTC m=+4228.029140618"
Sep 08 14:26:36.816931 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:36.816889    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:40.906359 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:40.906327    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:26:41.031816 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.031778    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-token\") pod \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") "
Sep 08 14:26:41.032008 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.031832    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-envoy\") pod \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") "
Sep 08 14:26:41.032008 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.031857    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-workload-certs\") pod \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") "
Sep 08 14:26:41.032008 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.031873    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-data\") pod \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") "
Sep 08 14:26:41.032008 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.031898    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-credential-socket\") pod \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") "
Sep 08 14:26:41.032008 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.031934    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istiod-ca-cert\") pod \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") "
Sep 08 14:26:41.032008 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.031986    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxbdq\" (UniqueName: \"kubernetes.io/projected/47d85102-7c7d-4aad-aa92-d9e41a7ea535-kube-api-access-wxbdq\") pod \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") "
Sep 08 14:26:41.032332 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.032043    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-workload-socket\") pod \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") "
Sep 08 14:26:41.032332 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.032091    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-podinfo\") pod \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\" (UID: \"47d85102-7c7d-4aad-aa92-d9e41a7ea535\") "
Sep 08 14:26:41.032332 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.032224    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "47d85102-7c7d-4aad-aa92-d9e41a7ea535" (UID: "47d85102-7c7d-4aad-aa92-d9e41a7ea535"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:26:41.032332 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.032284    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-data" (OuterVolumeSpecName: "istio-data") pod "47d85102-7c7d-4aad-aa92-d9e41a7ea535" (UID: "47d85102-7c7d-4aad-aa92-d9e41a7ea535"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:26:41.032611 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.032423    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "47d85102-7c7d-4aad-aa92-d9e41a7ea535" (UID: "47d85102-7c7d-4aad-aa92-d9e41a7ea535"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:26:41.032611 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.032438    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "47d85102-7c7d-4aad-aa92-d9e41a7ea535" (UID: "47d85102-7c7d-4aad-aa92-d9e41a7ea535"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:26:41.032611 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.032449    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-workload-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:26:41.032611 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.032471    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-data\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:26:41.032611 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.032530    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "47d85102-7c7d-4aad-aa92-d9e41a7ea535" (UID: "47d85102-7c7d-4aad-aa92-d9e41a7ea535"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:26:41.034258 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.034230    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-token" (OuterVolumeSpecName: "istio-token") pod "47d85102-7c7d-4aad-aa92-d9e41a7ea535" (UID: "47d85102-7c7d-4aad-aa92-d9e41a7ea535"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:26:41.034258 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.034246    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d85102-7c7d-4aad-aa92-d9e41a7ea535-kube-api-access-wxbdq" (OuterVolumeSpecName: "kube-api-access-wxbdq") pod "47d85102-7c7d-4aad-aa92-d9e41a7ea535" (UID: "47d85102-7c7d-4aad-aa92-d9e41a7ea535"). InnerVolumeSpecName "kube-api-access-wxbdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:26:41.034450 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.034353    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "47d85102-7c7d-4aad-aa92-d9e41a7ea535" (UID: "47d85102-7c7d-4aad-aa92-d9e41a7ea535"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:26:41.034450 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.034432    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "47d85102-7c7d-4aad-aa92-d9e41a7ea535" (UID: "47d85102-7c7d-4aad-aa92-d9e41a7ea535"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 14:26:41.133133 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.133096    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-credential-socket\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:26:41.133133 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.133130    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istiod-ca-cert\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:26:41.133133 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.133140    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxbdq\" (UniqueName: \"kubernetes.io/projected/47d85102-7c7d-4aad-aa92-d9e41a7ea535-kube-api-access-wxbdq\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:26:41.133133 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.133149    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-workload-socket\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:26:41.133443 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.133158    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-podinfo\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:26:41.133443 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.133166    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-token\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:26:41.133443 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.133174    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d85102-7c7d-4aad-aa92-d9e41a7ea535-istio-envoy\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:26:41.561339 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.561302    2561 generic.go:356] "Generic (PLEG): container finished" podID="47d85102-7c7d-4aad-aa92-d9e41a7ea535" containerID="c6ebb25b0db2a8776fe61137ff160bc9b24ea0610b5f423632aa2962421a837b" exitCode=0
Sep 08 14:26:41.561517 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.561363    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"
Sep 08 14:26:41.561517 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.561392    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn" event={"ID":"47d85102-7c7d-4aad-aa92-d9e41a7ea535","Type":"ContainerDied","Data":"c6ebb25b0db2a8776fe61137ff160bc9b24ea0610b5f423632aa2962421a837b"}
Sep 08 14:26:41.561517 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.561448    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn" event={"ID":"47d85102-7c7d-4aad-aa92-d9e41a7ea535","Type":"ContainerDied","Data":"e476505f7e1dd6734935cbc72c93996ba43531ee22453a0f7a1fddbb70ea7e57"}
Sep 08 14:26:41.561517 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.561472    2561 scope.go:117] "RemoveContainer" containerID="c6ebb25b0db2a8776fe61137ff160bc9b24ea0610b5f423632aa2962421a837b"
Sep 08 14:26:41.571027 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.571009    2561 scope.go:117] "RemoveContainer" containerID="b8cd9244374d1b9d662108b7e32cb0f5d5520708b648b28ce60b9d706ddead17"
Sep 08 14:26:41.579748 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.579723    2561 scope.go:117] "RemoveContainer" containerID="c6ebb25b0db2a8776fe61137ff160bc9b24ea0610b5f423632aa2962421a837b"
Sep 08 14:26:41.580074 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:26:41.580051    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ebb25b0db2a8776fe61137ff160bc9b24ea0610b5f423632aa2962421a837b\": container with ID starting with c6ebb25b0db2a8776fe61137ff160bc9b24ea0610b5f423632aa2962421a837b not found: ID does not exist" containerID="c6ebb25b0db2a8776fe61137ff160bc9b24ea0610b5f423632aa2962421a837b"
Sep 08 14:26:41.580124 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.580081    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ebb25b0db2a8776fe61137ff160bc9b24ea0610b5f423632aa2962421a837b"} err="failed to get container status \"c6ebb25b0db2a8776fe61137ff160bc9b24ea0610b5f423632aa2962421a837b\": rpc error: code = NotFound desc = could not find container \"c6ebb25b0db2a8776fe61137ff160bc9b24ea0610b5f423632aa2962421a837b\": container with ID starting with c6ebb25b0db2a8776fe61137ff160bc9b24ea0610b5f423632aa2962421a837b not found: ID does not exist"
Sep 08 14:26:41.580124 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.580100    2561 scope.go:117] "RemoveContainer" containerID="b8cd9244374d1b9d662108b7e32cb0f5d5520708b648b28ce60b9d706ddead17"
Sep 08 14:26:41.580391 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:26:41.580373    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8cd9244374d1b9d662108b7e32cb0f5d5520708b648b28ce60b9d706ddead17\": container with ID starting with b8cd9244374d1b9d662108b7e32cb0f5d5520708b648b28ce60b9d706ddead17 not found: ID does not exist" containerID="b8cd9244374d1b9d662108b7e32cb0f5d5520708b648b28ce60b9d706ddead17"
Sep 08 14:26:41.580457 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.580395    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8cd9244374d1b9d662108b7e32cb0f5d5520708b648b28ce60b9d706ddead17"} err="failed to get container status \"b8cd9244374d1b9d662108b7e32cb0f5d5520708b648b28ce60b9d706ddead17\": rpc error: code = NotFound desc = could not find container \"b8cd9244374d1b9d662108b7e32cb0f5d5520708b648b28ce60b9d706ddead17\": container with ID starting with b8cd9244374d1b9d662108b7e32cb0f5d5520708b648b28ce60b9d706ddead17 not found: ID does not exist"
Sep 08 14:26:41.584765 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.584739    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"]
Sep 08 14:26:41.589871 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:41.589845    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-4vkxn"]
Sep 08 14:26:43.007087 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:43.007052    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47d85102-7c7d-4aad-aa92-d9e41a7ea535" path="/var/lib/kubelet/pods/47d85102-7c7d-4aad-aa92-d9e41a7ea535/volumes"
Sep 08 14:26:45.816818 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:45.816783    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:45.829438 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:45.829407    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:26:46.600535 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:26:46.600503    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:27:07.533443 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:07.533411    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w"
Sep 08 14:27:30.574326 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:30.574277    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"]
Sep 08 14:27:30.581007 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:30.580927    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w"]
Sep 08 14:27:30.584966 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:27:30.584927    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 14:27:30.584966 ip-10-0-130-178 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 14:27:30.584966 ip-10-0-130-178 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 08 14:27:30.584966 ip-10-0-130-178 kubenswrapper[2561]:  > containerID="043015e38712a29db94316de315731ff9d96b9d3513ee4729d0eebcd29857d08" cmd=["/bin/sleep","15"]
Sep 08 14:27:30.585419 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:30.585389    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6" message="" err=<
Sep 08 14:27:30.585419 ip-10-0-130-178 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 14:27:30.585419 ip-10-0-130-178 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 08 14:27:30.585419 ip-10-0-130-178 kubenswrapper[2561]:  >
Sep 08 14:27:30.585644 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:27:30.585420    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 14:27:30.585644 ip-10-0-130-178 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 14:27:30.585644 ip-10-0-130-178 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 08 14:27:30.585644 ip-10-0-130-178 kubenswrapper[2561]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6" podUID="1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910" containerName="main" containerID="cri-o://043015e38712a29db94316de315731ff9d96b9d3513ee4729d0eebcd29857d08"
Sep 08 14:27:30.585644 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:30.585464    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6" podUID="1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910" containerName="main" containerID="cri-o://043015e38712a29db94316de315731ff9d96b9d3513ee4729d0eebcd29857d08" gracePeriod=60
Sep 08 14:27:30.589591 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:30.589559    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6" podUID="1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 08 14:27:30.836465 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:30.836372    2561 generic.go:356] "Generic (PLEG): container finished" podID="1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910" containerID="043015e38712a29db94316de315731ff9d96b9d3513ee4729d0eebcd29857d08" exitCode=0
Sep 08 14:27:30.836465 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:30.836450    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6" event={"ID":"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910","Type":"ContainerDied","Data":"043015e38712a29db94316de315731ff9d96b9d3513ee4729d0eebcd29857d08"}
Sep 08 14:27:31.339333 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.339307    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:27:31.493469 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.493384    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-home\") pod \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") "
Sep 08 14:27:31.493469 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.493427    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-model-cache\") pod \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") "
Sep 08 14:27:31.493469 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.493466    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-dshm\") pod \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") "
Sep 08 14:27:31.493732 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.493490    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l76t8\" (UniqueName: \"kubernetes.io/projected/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-kube-api-access-l76t8\") pod \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") "
Sep 08 14:27:31.493732 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.493525    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-tls-certs\") pod \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") "
Sep 08 14:27:31.493732 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.493561    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-tmp-dir\") pod \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\" (UID: \"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910\") "
Sep 08 14:27:31.493732 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.493690    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-home" (OuterVolumeSpecName: "home") pod "1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910" (UID: "1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:27:31.493732 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.493707    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-model-cache" (OuterVolumeSpecName: "model-cache") pod "1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910" (UID: "1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:27:31.493980 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.493848    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-home\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:27:31.493980 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.493869    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-model-cache\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:27:31.493980 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.493886    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910" (UID: "1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:27:31.495903 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.495878    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910" (UID: "1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:27:31.496012 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.495914    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-dshm" (OuterVolumeSpecName: "dshm") pod "1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910" (UID: "1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:27:31.496012 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.495927    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-kube-api-access-l76t8" (OuterVolumeSpecName: "kube-api-access-l76t8") pod "1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910" (UID: "1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910"). InnerVolumeSpecName "kube-api-access-l76t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:27:31.594530 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.594494    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:27:31.594530 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.594522    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-tmp-dir\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:27:31.594530 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.594531    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-dshm\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:27:31.594965 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.594540    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l76t8\" (UniqueName: \"kubernetes.io/projected/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910-kube-api-access-l76t8\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:27:31.842862 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.842774    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"
Sep 08 14:27:31.842862 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.842796    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6" event={"ID":"1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910","Type":"ContainerDied","Data":"58a4fb61bf596934734ba8448dacf72e35fd34cce036811b900b9c584ce76eb7"}
Sep 08 14:27:31.842862 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.842850    2561 scope.go:117] "RemoveContainer" containerID="043015e38712a29db94316de315731ff9d96b9d3513ee4729d0eebcd29857d08"
Sep 08 14:27:31.868457 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.868429    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"]
Sep 08 14:27:31.873697 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:31.873673    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-h6bd6"]
Sep 08 14:27:33.006199 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:33.006161    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910" path="/var/lib/kubelet/pods/1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910/volumes"
Sep 08 14:27:37.533545 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:37.533503    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w" podUID="347d5927-f0c3-46fb-8ac7-4971f7f0f786" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:27:45.606333 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:45.606276    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w" podUID="347d5927-f0c3-46fb-8ac7-4971f7f0f786" containerName="main" containerID="cri-o://dbaab007ef4f6ebec519d99217f1546d5b990c9b2b498cde05aaf575bf9c08fa" gracePeriod=45
Sep 08 14:27:47.533162 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:47.533112    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w" podUID="347d5927-f0c3-46fb-8ac7-4971f7f0f786" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:27:57.533201 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:57.533157    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w" podUID="347d5927-f0c3-46fb-8ac7-4971f7f0f786" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:27:57.533682 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:27:57.533266    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w"
Sep 08 14:28:07.533362 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:07.533254    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w" podUID="347d5927-f0c3-46fb-8ac7-4971f7f0f786" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 14:28:15.854429 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:15.854398    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w"
Sep 08 14:28:15.979028 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:15.978991    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsqb4\" (UniqueName: \"kubernetes.io/projected/347d5927-f0c3-46fb-8ac7-4971f7f0f786-kube-api-access-tsqb4\") pod \"347d5927-f0c3-46fb-8ac7-4971f7f0f786\" (UID: \"347d5927-f0c3-46fb-8ac7-4971f7f0f786\") "
Sep 08 14:28:15.979199 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:15.979125    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347d5927-f0c3-46fb-8ac7-4971f7f0f786-tls-certs\") pod \"347d5927-f0c3-46fb-8ac7-4971f7f0f786\" (UID: \"347d5927-f0c3-46fb-8ac7-4971f7f0f786\") "
Sep 08 14:28:15.981231 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:15.981206    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347d5927-f0c3-46fb-8ac7-4971f7f0f786-kube-api-access-tsqb4" (OuterVolumeSpecName: "kube-api-access-tsqb4") pod "347d5927-f0c3-46fb-8ac7-4971f7f0f786" (UID: "347d5927-f0c3-46fb-8ac7-4971f7f0f786"). InnerVolumeSpecName "kube-api-access-tsqb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:28:15.981311 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:15.981246    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/347d5927-f0c3-46fb-8ac7-4971f7f0f786-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "347d5927-f0c3-46fb-8ac7-4971f7f0f786" (UID: "347d5927-f0c3-46fb-8ac7-4971f7f0f786"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:28:16.065629 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:16.065591    2561 generic.go:356] "Generic (PLEG): container finished" podID="347d5927-f0c3-46fb-8ac7-4971f7f0f786" containerID="dbaab007ef4f6ebec519d99217f1546d5b990c9b2b498cde05aaf575bf9c08fa" exitCode=0
Sep 08 14:28:16.065825 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:16.065640    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w"
Sep 08 14:28:16.065825 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:16.065672    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w" event={"ID":"347d5927-f0c3-46fb-8ac7-4971f7f0f786","Type":"ContainerDied","Data":"dbaab007ef4f6ebec519d99217f1546d5b990c9b2b498cde05aaf575bf9c08fa"}
Sep 08 14:28:16.065825 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:16.065723    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w" event={"ID":"347d5927-f0c3-46fb-8ac7-4971f7f0f786","Type":"ContainerDied","Data":"76114e084b9b614feba03cb0cba0501cf6ef8ac006f4a1d6f37ebb1fbacc16bf"}
Sep 08 14:28:16.065825 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:16.065745    2561 scope.go:117] "RemoveContainer" containerID="dbaab007ef4f6ebec519d99217f1546d5b990c9b2b498cde05aaf575bf9c08fa"
Sep 08 14:28:16.075134 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:16.075116    2561 scope.go:117] "RemoveContainer" containerID="dbaab007ef4f6ebec519d99217f1546d5b990c9b2b498cde05aaf575bf9c08fa"
Sep 08 14:28:16.075446 ip-10-0-130-178 kubenswrapper[2561]: E0908 14:28:16.075428    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbaab007ef4f6ebec519d99217f1546d5b990c9b2b498cde05aaf575bf9c08fa\": container with ID starting with dbaab007ef4f6ebec519d99217f1546d5b990c9b2b498cde05aaf575bf9c08fa not found: ID does not exist" containerID="dbaab007ef4f6ebec519d99217f1546d5b990c9b2b498cde05aaf575bf9c08fa"
Sep 08 14:28:16.075506 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:16.075455    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbaab007ef4f6ebec519d99217f1546d5b990c9b2b498cde05aaf575bf9c08fa"} err="failed to get container status \"dbaab007ef4f6ebec519d99217f1546d5b990c9b2b498cde05aaf575bf9c08fa\": rpc error: code = NotFound desc = could not find container \"dbaab007ef4f6ebec519d99217f1546d5b990c9b2b498cde05aaf575bf9c08fa\": container with ID starting with dbaab007ef4f6ebec519d99217f1546d5b990c9b2b498cde05aaf575bf9c08fa not found: ID does not exist"
Sep 08 14:28:16.080306 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:16.080269    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347d5927-f0c3-46fb-8ac7-4971f7f0f786-tls-certs\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:28:16.080418 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:16.080315    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsqb4\" (UniqueName: \"kubernetes.io/projected/347d5927-f0c3-46fb-8ac7-4971f7f0f786-kube-api-access-tsqb4\") on node \"ip-10-0-130-178.ec2.internal\" DevicePath \"\""
Sep 08 14:28:16.086605 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:16.086583    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w"]
Sep 08 14:28:16.089345 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:16.089323    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cfk6w"]
Sep 08 14:28:17.006915 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:17.006884    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="347d5927-f0c3-46fb-8ac7-4971f7f0f786" path="/var/lib/kubelet/pods/347d5927-f0c3-46fb-8ac7-4971f7f0f786/volumes"
Sep 08 14:28:44.660618 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:44.660579    2561 ???:1] "http: TLS handshake error from 10.0.130.178:47328: EOF"
Sep 08 14:28:44.856793 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:44.856760    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-g49x5_1def0316-87d7-4d12-b8ec-739a5dbe6f1a/discovery/0.log"
Sep 08 14:28:44.880732 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:44.880698    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-b5cvn_db8dab86-bf9e-4760-8002-d3a068f8d45d/istio-proxy/0.log"
Sep 08 14:28:44.911962 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:44.911848    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-b5cvn_db8dab86-bf9e-4760-8002-d3a068f8d45d/istio-proxy/1.log"
Sep 08 14:28:45.816648 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:45.816614    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-g49x5_1def0316-87d7-4d12-b8ec-739a5dbe6f1a/discovery/0.log"
Sep 08 14:28:45.842100 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:45.842071    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-b5cvn_db8dab86-bf9e-4760-8002-d3a068f8d45d/istio-proxy/0.log"
Sep 08 14:28:45.870592 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:45.870558    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-b5cvn_db8dab86-bf9e-4760-8002-d3a068f8d45d/istio-proxy/1.log"
Sep 08 14:28:47.056842 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:47.056794    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-znxvr_b2dc2c9e-aec8-4eb7-a237-4ad984cb56be/manager/0.log"
Sep 08 14:28:47.251520 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:47.251479    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-vgnf6_d039964a-035a-4e2f-bbc9-6a1aabf1d7be/manager/0.log"
Sep 08 14:28:55.090249 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:55.090211    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ktjbq_1a17d5ad-fbe7-4387-aa74-57dbe6e2f03e/global-pull-secret-syncer/0.log"
Sep 08 14:28:55.302184 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:55.302150    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ql8vc_53ffac54-50e1-4c1e-97b1-b18f1164f56d/konnectivity-agent/0.log"
Sep 08 14:28:55.391929 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:28:55.391851    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-178.ec2.internal_b0b12b2fbd9460cffd368b1bb1ef73f5/haproxy/0.log"
Sep 08 14:29:00.990109 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:00.990075    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-znxvr_b2dc2c9e-aec8-4eb7-a237-4ad984cb56be/manager/0.log"
Sep 08 14:29:01.341204 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:01.341107    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-vgnf6_d039964a-035a-4e2f-bbc9-6a1aabf1d7be/manager/0.log"
Sep 08 14:29:03.317307 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:03.317253    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-dj7cl_773cfcaf-f677-4056-bbe8-1a475443aa1b/cluster-monitoring-operator/0.log"
Sep 08 14:29:03.365608 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:03.365574    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-fhrnx_b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986/kube-state-metrics/0.log"
Sep 08 14:29:03.435497 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:03.435469    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-fhrnx_b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986/kube-rbac-proxy-main/0.log"
Sep 08 14:29:03.466049 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:03.466021    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-fhrnx_b0cb6c04-2dc0-4a79-83f5-cb6fc6c5e986/kube-rbac-proxy-self/0.log"
Sep 08 14:29:03.879036 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:03.879005    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q7vd2_5b281d14-753f-4ac3-a725-ce90bca29a25/node-exporter/0.log"
Sep 08 14:29:03.916256 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:03.916225    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q7vd2_5b281d14-753f-4ac3-a725-ce90bca29a25/kube-rbac-proxy/0.log"
Sep 08 14:29:03.946599 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:03.946571    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q7vd2_5b281d14-753f-4ac3-a725-ce90bca29a25/init-textfile/0.log"
Sep 08 14:29:03.983805 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:03.983777    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-df4nq_8ce9ed2a-e752-42d2-9c83-88ee0c17532d/kube-rbac-proxy-main/0.log"
Sep 08 14:29:04.035299 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.035269    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-df4nq_8ce9ed2a-e752-42d2-9c83-88ee0c17532d/kube-rbac-proxy-self/0.log"
Sep 08 14:29:04.075675 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.075646    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-df4nq_8ce9ed2a-e752-42d2-9c83-88ee0c17532d/openshift-state-metrics/0.log"
Sep 08 14:29:04.869766 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.869727    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"]
Sep 08 14:29:04.870164 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.870065    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="347d5927-f0c3-46fb-8ac7-4971f7f0f786" containerName="main"
Sep 08 14:29:04.870164 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.870076    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="347d5927-f0c3-46fb-8ac7-4971f7f0f786" containerName="main"
Sep 08 14:29:04.870164 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.870088    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47d85102-7c7d-4aad-aa92-d9e41a7ea535" containerName="istio-proxy"
Sep 08 14:29:04.870164 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.870094    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="47d85102-7c7d-4aad-aa92-d9e41a7ea535" containerName="istio-proxy"
Sep 08 14:29:04.870164 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.870107    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47d85102-7c7d-4aad-aa92-d9e41a7ea535" containerName="istio-proxy"
Sep 08 14:29:04.870164 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.870122    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="47d85102-7c7d-4aad-aa92-d9e41a7ea535" containerName="istio-proxy"
Sep 08 14:29:04.870164 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.870130    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910" containerName="main"
Sep 08 14:29:04.870164 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.870135    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910" containerName="main"
Sep 08 14:29:04.870493 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.870214    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="347d5927-f0c3-46fb-8ac7-4971f7f0f786" containerName="main"
Sep 08 14:29:04.870493 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.870222    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bb5bc8a-bf24-4bc5-bca4-9f1a9736e910" containerName="main"
Sep 08 14:29:04.870493 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.870229    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="47d85102-7c7d-4aad-aa92-d9e41a7ea535" containerName="istio-proxy"
Sep 08 14:29:04.870493 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.870234    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="47d85102-7c7d-4aad-aa92-d9e41a7ea535" containerName="istio-proxy"
Sep 08 14:29:04.873224 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.873205    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"
Sep 08 14:29:04.876233 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.876212    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nm4b8\"/\"openshift-service-ca.crt\""
Sep 08 14:29:04.876689 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.876672    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nm4b8\"/\"default-dockercfg-fsrkr\""
Sep 08 14:29:04.876987 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.876967    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nm4b8\"/\"kube-root-ca.crt\""
Sep 08 14:29:04.907563 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.907529    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"]
Sep 08 14:29:04.999648 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.999607    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/44d3b808-d660-4959-846d-1126859c4d52-proc\") pod \"perf-node-gather-daemonset-dfskr\" (UID: \"44d3b808-d660-4959-846d-1126859c4d52\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"
Sep 08 14:29:04.999648 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.999654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44d3b808-d660-4959-846d-1126859c4d52-sys\") pod \"perf-node-gather-daemonset-dfskr\" (UID: \"44d3b808-d660-4959-846d-1126859c4d52\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"
Sep 08 14:29:04.999868 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.999737    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/44d3b808-d660-4959-846d-1126859c4d52-podres\") pod \"perf-node-gather-daemonset-dfskr\" (UID: \"44d3b808-d660-4959-846d-1126859c4d52\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"
Sep 08 14:29:04.999868 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.999777    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44d3b808-d660-4959-846d-1126859c4d52-lib-modules\") pod \"perf-node-gather-daemonset-dfskr\" (UID: \"44d3b808-d660-4959-846d-1126859c4d52\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"
Sep 08 14:29:04.999868 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:04.999798    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkvsb\" (UniqueName: \"kubernetes.io/projected/44d3b808-d660-4959-846d-1126859c4d52-kube-api-access-jkvsb\") pod \"perf-node-gather-daemonset-dfskr\" (UID: \"44d3b808-d660-4959-846d-1126859c4d52\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"
Sep 08 14:29:05.101242 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:05.101201    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/44d3b808-d660-4959-846d-1126859c4d52-podres\") pod \"perf-node-gather-daemonset-dfskr\" (UID: \"44d3b808-d660-4959-846d-1126859c4d52\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"
Sep 08 14:29:05.101242 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:05.101259    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44d3b808-d660-4959-846d-1126859c4d52-lib-modules\") pod \"perf-node-gather-daemonset-dfskr\" (UID: \"44d3b808-d660-4959-846d-1126859c4d52\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"
Sep 08 14:29:05.101541 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:05.101317    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkvsb\" (UniqueName: \"kubernetes.io/projected/44d3b808-d660-4959-846d-1126859c4d52-kube-api-access-jkvsb\") pod \"perf-node-gather-daemonset-dfskr\" (UID: \"44d3b808-d660-4959-846d-1126859c4d52\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"
Sep 08 14:29:05.101541 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:05.101411    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/44d3b808-d660-4959-846d-1126859c4d52-podres\") pod \"perf-node-gather-daemonset-dfskr\" (UID: \"44d3b808-d660-4959-846d-1126859c4d52\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"
Sep 08 14:29:05.101541 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:05.101428    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/44d3b808-d660-4959-846d-1126859c4d52-proc\") pod \"perf-node-gather-daemonset-dfskr\" (UID: \"44d3b808-d660-4959-846d-1126859c4d52\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"
Sep 08 14:29:05.101541 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:05.101466    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44d3b808-d660-4959-846d-1126859c4d52-lib-modules\") pod \"perf-node-gather-daemonset-dfskr\" (UID: \"44d3b808-d660-4959-846d-1126859c4d52\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"
Sep 08 14:29:05.101541 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:05.101474    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44d3b808-d660-4959-846d-1126859c4d52-sys\") pod \"perf-node-gather-daemonset-dfskr\" (UID: \"44d3b808-d660-4959-846d-1126859c4d52\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"
Sep 08 14:29:05.101541 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:05.101504    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44d3b808-d660-4959-846d-1126859c4d52-sys\") pod \"perf-node-gather-daemonset-dfskr\" (UID: \"44d3b808-d660-4959-846d-1126859c4d52\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"
Sep 08 14:29:05.101815 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:05.101569    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/44d3b808-d660-4959-846d-1126859c4d52-proc\") pod \"perf-node-gather-daemonset-dfskr\" (UID: \"44d3b808-d660-4959-846d-1126859c4d52\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"
Sep 08 14:29:05.115720 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:05.115691    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkvsb\" (UniqueName: \"kubernetes.io/projected/44d3b808-d660-4959-846d-1126859c4d52-kube-api-access-jkvsb\") pod \"perf-node-gather-daemonset-dfskr\" (UID: \"44d3b808-d660-4959-846d-1126859c4d52\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"
Sep 08 14:29:05.183223 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:05.183125    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"
Sep 08 14:29:05.470147 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:05.470112    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"]
Sep 08 14:29:05.472562 ip-10-0-130-178 kubenswrapper[2561]: W0908 14:29:05.472532    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod44d3b808_d660_4959_846d_1126859c4d52.slice/crio-b06770fcadbd783375c3f0eff144e3195e2b64a321cb411768b874a25d6c54da WatchSource:0}: Error finding container b06770fcadbd783375c3f0eff144e3195e2b64a321cb411768b874a25d6c54da: Status 404 returned error can't find the container with id b06770fcadbd783375c3f0eff144e3195e2b64a321cb411768b874a25d6c54da
Sep 08 14:29:06.323814 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:06.323776    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr" event={"ID":"44d3b808-d660-4959-846d-1126859c4d52","Type":"ContainerStarted","Data":"a5ecdcd4cb596904a68b6a4ae676a14459d841b8135dfa9eca19253f454d0f67"}
Sep 08 14:29:06.324250 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:06.323832    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr" event={"ID":"44d3b808-d660-4959-846d-1126859c4d52","Type":"ContainerStarted","Data":"b06770fcadbd783375c3f0eff144e3195e2b64a321cb411768b874a25d6c54da"}
Sep 08 14:29:06.324250 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:06.323879    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"
Sep 08 14:29:06.347548 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:06.347496    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr" podStartSLOduration=2.347480779 podStartE2EDuration="2.347480779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:29:06.343868358 +0000 UTC m=+4377.804513165" watchObservedRunningTime="2026-09-08 14:29:06.347480779 +0000 UTC m=+4377.808125589"
Sep 08 14:29:06.797545 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:06.797515    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/1.log"
Sep 08 14:29:06.802598 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:06.802570    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-6g876_5e21ee11-3e84-4622-8cc7-4984c76637aa/console-operator/2.log"
Sep 08 14:29:07.391367 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:07.391339    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-577596d6f-fsg59_af47a1c7-099c-4fb1-9f65-3beb22ed88ab/console/0.log"
Sep 08 14:29:08.038985 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:08.038950    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-lmphg_bc3bc449-a85e-4f82-bede-62b391eac326/volume-data-source-validator/0.log"
Sep 08 14:29:09.192000 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:09.191969    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-242sh_dcfe8764-419e-438f-91fb-c2f3892df1b2/dns/0.log"
Sep 08 14:29:09.222030 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:09.222002    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-242sh_dcfe8764-419e-438f-91fb-c2f3892df1b2/kube-rbac-proxy/0.log"
Sep 08 14:29:09.453105 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:09.453019    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t8k5r_bb02bae4-4bf7-4018-b0a3-e9f65b5057fb/dns-node-resolver/0.log"
Sep 08 14:29:10.057131 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:10.057038    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-b96b98575-gslm8_c21fa545-edea-4e71-b403-f68625c83518/registry/0.log"
Sep 08 14:29:10.141583 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:10.141546    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f4gjd_fca118ca-6b4f-4c1b-9fa7-b384f31387a4/node-ca/0.log"
Sep 08 14:29:11.295134 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:11.295100    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-g49x5_1def0316-87d7-4d12-b8ec-739a5dbe6f1a/discovery/0.log"
Sep 08 14:29:11.327707 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:11.327675    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-b5cvn_db8dab86-bf9e-4760-8002-d3a068f8d45d/istio-proxy/0.log"
Sep 08 14:29:11.355639 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:11.355607    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-b5cvn_db8dab86-bf9e-4760-8002-d3a068f8d45d/istio-proxy/1.log"
Sep 08 14:29:12.007775 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:12.007742    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q5qrk_a54b8b62-55df-4e1c-a738-ae5081203413/serve-healthcheck-canary/0.log"
Sep 08 14:29:12.339578 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:12.339496    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-dfskr"
Sep 08 14:29:12.642005 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:12.641927    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hhmrh_c20793a7-cc59-453b-9942-d527170ce508/kube-rbac-proxy/0.log"
Sep 08 14:29:12.664529 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:12.664498    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hhmrh_c20793a7-cc59-453b-9942-d527170ce508/exporter/0.log"
Sep 08 14:29:12.689105 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:12.689077    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hhmrh_c20793a7-cc59-453b-9942-d527170ce508/extractor/0.log"
Sep 08 14:29:15.456727 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:15.456698    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-g49x5_1def0316-87d7-4d12-b8ec-739a5dbe6f1a/discovery/0.log"
Sep 08 14:29:15.473023 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:15.472995    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-b5cvn_db8dab86-bf9e-4760-8002-d3a068f8d45d/istio-proxy/0.log"
Sep 08 14:29:15.503636 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:15.503603    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-b5cvn_db8dab86-bf9e-4760-8002-d3a068f8d45d/istio-proxy/1.log"
Sep 08 14:29:16.728770 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:16.728735    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-g49x5_1def0316-87d7-4d12-b8ec-739a5dbe6f1a/discovery/0.log"
Sep 08 14:29:16.762839 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:16.762807    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-b5cvn_db8dab86-bf9e-4760-8002-d3a068f8d45d/istio-proxy/0.log"
Sep 08 14:29:16.790825 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:16.790793    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-b5cvn_db8dab86-bf9e-4760-8002-d3a068f8d45d/istio-proxy/1.log"
Sep 08 14:29:17.522596 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:17.522566    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-575d6764f4-tw7r2_c713585e-70c3-4be8-a1d5-91b521dc205a/manager/0.log"
Sep 08 14:29:17.884010 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:17.883920    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-945498d87-2sh5b_faaa3499-807c-436d-a97b-ae134b995040/manager/0.log"
Sep 08 14:29:17.965536 ip-10-0-130-178 kubenswrapper[2561]: I0908 14:29:17.965508    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-9hzrh_511f6abd-9084-449c-a2fb-4c4035463892/server/0.log"