Apr 22 19:55:18.862139 ip-10-0-135-245 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Apr 22 19:55:18.862152 ip-10-0-135-245 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Apr 22 19:55:18.862162 ip-10-0-135-245 systemd[1]: kubelet.service: Failed with result 'resources'.
Apr 22 19:55:18.862498 ip-10-0-135-245 systemd[1]: Failed to start Kubernetes Kubelet.
Apr 22 19:55:29.887275 ip-10-0-135-245 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Apr 22 19:55:29.887293 ip-10-0-135-245 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 97a3a66c7b5d4fe1809226ab10caa733 --
Apr 22 19:57:44.796436 ip-10-0-135-245 systemd[1]: Starting Kubernetes Kubelet...
Apr 22 19:57:45.167719 ip-10-0-135-245 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 22 19:57:45.167719 ip-10-0-135-245 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Apr 22 19:57:45.167719 ip-10-0-135-245 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 22 19:57:45.167719 ip-10-0-135-245 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Apr 22 19:57:45.167719 ip-10-0-135-245 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 22 19:57:45.169925 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.169835    2576 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Apr 22 19:57:45.176363 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176341    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 22 19:57:45.176363 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176359    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 22 19:57:45.176363 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176363    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 22 19:57:45.176363 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176366    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 22 19:57:45.176363 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176369    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 22 19:57:45.176556 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176373    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 22 19:57:45.176556 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176377    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 22 19:57:45.176556 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176380    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 22 19:57:45.176556 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176383    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 22 19:57:45.176556 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176385    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 22 19:57:45.176556 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176388    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 22 19:57:45.176556 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176391    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 22 19:57:45.176556 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176394    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 22 19:57:45.176556 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176397    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 22 19:57:45.176556 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176399    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 22 19:57:45.176556 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176402    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 22 19:57:45.176556 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176405    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 22 19:57:45.176556 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176407    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 22 19:57:45.176556 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176410    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 22 19:57:45.176556 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176412    2576 feature_gate.go:328] unrecognized feature gate: Example2
Apr 22 19:57:45.176556 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176415    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 22 19:57:45.176556 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176417    2576 feature_gate.go:328] unrecognized feature gate: Example
Apr 22 19:57:45.176556 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176420    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 22 19:57:45.176556 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176422    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 22 19:57:45.176556 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176425    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 22 19:57:45.177028 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176428    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 22 19:57:45.177028 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176431    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 22 19:57:45.177028 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176433    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 22 19:57:45.177028 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176436    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 22 19:57:45.177028 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176438    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 22 19:57:45.177028 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176441    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 22 19:57:45.177028 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176443    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 22 19:57:45.177028 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176446    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 22 19:57:45.177028 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176450    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 22 19:57:45.177028 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176453    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 22 19:57:45.177028 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176456    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 22 19:57:45.177028 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176459    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 22 19:57:45.177028 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176462    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 22 19:57:45.177028 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176466    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 22 19:57:45.177028 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176468    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 22 19:57:45.177028 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176471    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 22 19:57:45.177028 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176474    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 22 19:57:45.177028 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176476    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 22 19:57:45.177028 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176478    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 22 19:57:45.177513 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176481    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 22 19:57:45.177513 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176484    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 22 19:57:45.177513 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176486    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 22 19:57:45.177513 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176488    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 22 19:57:45.177513 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176492    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 22 19:57:45.177513 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176494    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 22 19:57:45.177513 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176497    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 22 19:57:45.177513 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176499    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 22 19:57:45.177513 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176502    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 22 19:57:45.177513 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176504    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 22 19:57:45.177513 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176506    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 22 19:57:45.177513 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176509    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 22 19:57:45.177513 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176511    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 22 19:57:45.177513 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176521    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 22 19:57:45.177513 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176523    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 22 19:57:45.177513 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176526    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 22 19:57:45.177513 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176528    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 22 19:57:45.177513 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176530    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 22 19:57:45.177513 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176533    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 22 19:57:45.177513 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176537    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 22 19:57:45.178014 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176539    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 22 19:57:45.178014 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176541    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 22 19:57:45.178014 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176544    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 22 19:57:45.178014 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176546    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 22 19:57:45.178014 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176549    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 22 19:57:45.178014 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176551    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 22 19:57:45.178014 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176556    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 22 19:57:45.178014 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176559    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 22 19:57:45.178014 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176562    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 22 19:57:45.178014 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176564    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 22 19:57:45.178014 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176568    2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 22 19:57:45.178014 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176571    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 22 19:57:45.178014 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176574    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 22 19:57:45.178014 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176577    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 22 19:57:45.178014 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176580    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 22 19:57:45.178014 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176582    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 22 19:57:45.178014 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176586    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 22 19:57:45.178014 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176589    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 22 19:57:45.178014 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176592    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 22 19:57:45.178497 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176595    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 22 19:57:45.178497 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176597    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 22 19:57:45.178497 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176600    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 22 19:57:45.178497 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176984    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 22 19:57:45.178497 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176989    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 22 19:57:45.178497 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176992    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 22 19:57:45.178497 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176995    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 22 19:57:45.178497 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.176998    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 22 19:57:45.178497 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177000    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 22 19:57:45.178497 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177003    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 22 19:57:45.178497 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177005    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 22 19:57:45.178497 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177008    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 22 19:57:45.178497 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177011    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 22 19:57:45.178497 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177013    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 22 19:57:45.178497 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177016    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 22 19:57:45.178497 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177019    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 22 19:57:45.178497 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177023    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 22 19:57:45.178497 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177025    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 22 19:57:45.178497 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177029    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 22 19:57:45.178497 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177032    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 22 19:57:45.178968 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177035    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 22 19:57:45.178968 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177037    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 22 19:57:45.178968 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177040    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 22 19:57:45.178968 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177043    2576 feature_gate.go:328] unrecognized feature gate: Example2
Apr 22 19:57:45.178968 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177046    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 22 19:57:45.178968 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177051    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 22 19:57:45.178968 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177054    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 22 19:57:45.178968 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177058    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 22 19:57:45.178968 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177061    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 22 19:57:45.178968 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177063    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 22 19:57:45.178968 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177066    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 22 19:57:45.178968 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177068    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 22 19:57:45.178968 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177071    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 22 19:57:45.178968 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177074    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 22 19:57:45.178968 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177076    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 22 19:57:45.178968 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177079    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 22 19:57:45.178968 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177082    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 22 19:57:45.178968 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177084    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 22 19:57:45.178968 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177087    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 22 19:57:45.178968 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177089    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 22 19:57:45.179475 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177092    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 22 19:57:45.179475 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177094    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 22 19:57:45.179475 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177097    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 22 19:57:45.179475 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177099    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 22 19:57:45.179475 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177102    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 22 19:57:45.179475 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177104    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 22 19:57:45.179475 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177106    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 22 19:57:45.179475 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177109    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 22 19:57:45.179475 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177112    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 22 19:57:45.179475 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177114    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 22 19:57:45.179475 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177116    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 22 19:57:45.179475 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177119    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 22 19:57:45.179475 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177121    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 22 19:57:45.179475 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177124    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 22 19:57:45.179475 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177127    2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 22 19:57:45.179475 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177129    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 22 19:57:45.179475 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177131    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 22 19:57:45.179475 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177134    2576 feature_gate.go:328] unrecognized feature gate: Example
Apr 22 19:57:45.179475 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177136    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 22 19:57:45.179475 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177139    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 22 19:57:45.179960 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177142    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 22 19:57:45.179960 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177159    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 22 19:57:45.179960 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177162    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 22 19:57:45.179960 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177165    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 22 19:57:45.179960 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177168    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 22 19:57:45.179960 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177170    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 22 19:57:45.179960 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177173    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 22 19:57:45.179960 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177175    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 22 19:57:45.179960 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177178    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 22 19:57:45.179960 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177180    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 22 19:57:45.179960 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177182    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 22 19:57:45.179960 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177185    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 22 19:57:45.179960 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177188    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 22 19:57:45.179960 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177191    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 22 19:57:45.179960 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177193    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 22 19:57:45.179960 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177196    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 22 19:57:45.179960 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177198    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 22 19:57:45.179960 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177201    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 22 19:57:45.179960 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177203    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 22 19:57:45.179960 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177206    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 22 19:57:45.180468 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177208    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 22 19:57:45.180468 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177211    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 22 19:57:45.180468 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177213    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 22 19:57:45.180468 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177216    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 22 19:57:45.180468 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177219    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 22 19:57:45.180468 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177221    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 22 19:57:45.180468 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177224    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 22 19:57:45.180468 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177226    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 22 19:57:45.180468 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.177229    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 22 19:57:45.180468 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178209    2576 flags.go:64] FLAG: --address="0.0.0.0"
Apr 22 19:57:45.180468 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178218    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Apr 22 19:57:45.180468 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178230    2576 flags.go:64] FLAG: --anonymous-auth="true"
Apr 22 19:57:45.180468 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178239    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Apr 22 19:57:45.180468 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178245    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Apr 22 19:57:45.180468 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178248    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Apr 22 19:57:45.180468 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178253    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Apr 22 19:57:45.180468 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178257    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Apr 22 19:57:45.180468 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178260    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Apr 22 19:57:45.180468 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178263    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Apr 22 19:57:45.180468 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178267    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Apr 22 19:57:45.180468 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178270    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178273    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178276    2576 flags.go:64] FLAG: --cgroup-root=""
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178279    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178282    2576 flags.go:64] FLAG: --client-ca-file=""
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178284    2576 flags.go:64] FLAG: --cloud-config=""
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178287    2576 flags.go:64] FLAG: --cloud-provider="external"
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178290    2576 flags.go:64] FLAG: --cluster-dns="[]"
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178294    2576 flags.go:64] FLAG: --cluster-domain=""
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178297    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178300    2576 flags.go:64] FLAG: --config-dir=""
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178303    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178306    2576 flags.go:64] FLAG: --container-log-max-files="5"
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178310    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178313    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178321    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178324    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178327    2576 flags.go:64] FLAG: --contention-profiling="false"
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178331    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178333    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178336    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178339    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178344    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178347    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178349    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Apr 22 19:57:45.180989 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178352    2576 flags.go:64] FLAG: --enable-load-reader="false"
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178356    2576 flags.go:64] FLAG: --enable-server="true"
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178358    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178363    2576 flags.go:64] FLAG: --event-burst="100"
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178366    2576 flags.go:64] FLAG: --event-qps="50"
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178369    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178372    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178375    2576 flags.go:64] FLAG: --eviction-hard=""
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178378    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178381    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178384    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178387    2576 flags.go:64] FLAG: --eviction-soft=""
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178390    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178394    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178397    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178400    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178403    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178406    2576 flags.go:64] FLAG: --fail-swap-on="true"
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178409    2576 flags.go:64] FLAG: --feature-gates=""
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178413    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178416    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178419    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178423    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178426    2576 flags.go:64] FLAG: --healthz-port="10248"
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178429    2576 flags.go:64] FLAG: --help="false"
Apr 22 19:57:45.181625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178432    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-135-245.ec2.internal"
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178435    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178438    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178441    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178444    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178447    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178450    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178453    2576 flags.go:64] FLAG: --image-service-endpoint=""
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178456    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178459    2576 flags.go:64] FLAG: --kube-api-burst="100"
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178462    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178465    2576 flags.go:64] FLAG: --kube-api-qps="50"
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178467    2576 flags.go:64] FLAG: --kube-reserved=""
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178470    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178473    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178476    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178479    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178481    2576 flags.go:64] FLAG: --lock-file=""
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178484    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178487    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178493    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178499    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178502    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178505    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Apr 22 19:57:45.182290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178508    2576 flags.go:64] FLAG: --logging-format="text"
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178510    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178514    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178517    2576 flags.go:64] FLAG: --manifest-url=""
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178520    2576 flags.go:64] FLAG: --manifest-url-header=""
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178526    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178529    2576 flags.go:64] FLAG: --max-open-files="1000000"
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178533    2576 flags.go:64] FLAG: --max-pods="110"
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178536    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178539    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178542    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178544    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178547    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178550    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178553    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178560    2576 flags.go:64] FLAG: --node-status-max-images="50"
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178563    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178566    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178570    2576 flags.go:64] FLAG: --pod-cidr=""
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178572    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8cfe89231412ff3ee8cb6207fa0be33cad0f08e88c9c0f1e9f7e8c6f14d6715"
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178578    2576 flags.go:64] FLAG: --pod-manifest-path=""
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178580    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178583    2576 flags.go:64] FLAG: --pods-per-core="0"
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178587    2576 flags.go:64] FLAG: --port="10250"
Apr 22 19:57:45.182860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178590    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178592    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ec5c265cca1d762c"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178595    2576 flags.go:64] FLAG: --qos-reserved=""
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178598    2576 flags.go:64] FLAG: --read-only-port="10255"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178602    2576 flags.go:64] FLAG: --register-node="true"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178605    2576 flags.go:64] FLAG: --register-schedulable="true"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178608    2576 flags.go:64] FLAG: --register-with-taints=""
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178611    2576 flags.go:64] FLAG: --registry-burst="10"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178614    2576 flags.go:64] FLAG: --registry-qps="5"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178617    2576 flags.go:64] FLAG: --reserved-cpus=""
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178619    2576 flags.go:64] FLAG: --reserved-memory=""
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178623    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178626    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178629    2576 flags.go:64] FLAG: --rotate-certificates="false"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178633    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178636    2576 flags.go:64] FLAG: --runonce="false"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178638    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178641    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178644    2576 flags.go:64] FLAG: --seccomp-default="false"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178647    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178650    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178653    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178656    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178659    2576 flags.go:64] FLAG: --storage-driver-password="root"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178661    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178664    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Apr 22 19:57:45.183439 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178667    2576 flags.go:64] FLAG: --storage-driver-user="root"
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178670    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178673    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178676    2576 flags.go:64] FLAG: --system-cgroups=""
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178679    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178684    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178687    2576 flags.go:64] FLAG: --tls-cert-file=""
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178690    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178694    2576 flags.go:64] FLAG: --tls-min-version=""
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178697    2576 flags.go:64] FLAG: --tls-private-key-file=""
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178701    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178704    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178707    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178710    2576 flags.go:64] FLAG: --v="2"
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178714    2576 flags.go:64] FLAG: --version="false"
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178718    2576 flags.go:64] FLAG: --vmodule=""
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178722    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.178725    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178815    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178819    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178823    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178826    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178829    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178832    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 22 19:57:45.184070 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178835    2576 feature_gate.go:328] unrecognized feature gate: Example2
Apr 22 19:57:45.184666 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178838    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 22 19:57:45.184666 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178840    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 22 19:57:45.184666 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178843    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 22 19:57:45.184666 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178846    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 22 19:57:45.184666 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178848    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 22 19:57:45.184666 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178851    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 22 19:57:45.184666 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178853    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 22 19:57:45.184666 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178856    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 22 19:57:45.184666 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178858    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 22 19:57:45.184666 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178861    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 22 19:57:45.184666 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178864    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 22 19:57:45.184666 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178866    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 22 19:57:45.184666 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178869    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 22 19:57:45.184666 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178872    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 22 19:57:45.184666 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178874    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 22 19:57:45.184666 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178876    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 22 19:57:45.184666 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178879    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 22 19:57:45.184666 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178883    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 22 19:57:45.184666 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178885    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 22 19:57:45.184666 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178888    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 22 19:57:45.185469 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178890    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 22 19:57:45.185469 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178893    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 22 19:57:45.185469 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178895    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 22 19:57:45.185469 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178898    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 22 19:57:45.185469 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178900    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 22 19:57:45.185469 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178902    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 22 19:57:45.185469 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178905    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 22 19:57:45.185469 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178908    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 22 19:57:45.185469 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178911    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 22 19:57:45.185469 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178913    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 22 19:57:45.185469 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178916    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 22 19:57:45.185469 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178918    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 22 19:57:45.185469 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178921    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 22 19:57:45.185469 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178923    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 22 19:57:45.185469 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178926    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 22 19:57:45.185469 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178928    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 22 19:57:45.185469 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178931    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 22 19:57:45.185469 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178933    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 22 19:57:45.185469 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178935    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 22 19:57:45.185469 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178938    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 22 19:57:45.186347 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178940    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 22 19:57:45.186347 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178943    2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 22 19:57:45.186347 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178946    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 22 19:57:45.186347 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178949    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 22 19:57:45.186347 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178953    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 22 19:57:45.186347 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178956    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 22 19:57:45.186347 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178958    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 22 19:57:45.186347 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178961    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 22 19:57:45.186347 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178963    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 22 19:57:45.186347 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178967    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 22 19:57:45.186347 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178969    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 22 19:57:45.186347 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178972    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 22 19:57:45.186347 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178974    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 22 19:57:45.186347 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178978    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 22 19:57:45.186347 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178982    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 22 19:57:45.186347 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178985    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 22 19:57:45.186347 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178988    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 22 19:57:45.186347 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178991    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 22 19:57:45.186347 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178993    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 22 19:57:45.187069 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.178997    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 22 19:57:45.187069 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.179000    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 22 19:57:45.187069 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.179002    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 22 19:57:45.187069 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.179005    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 22 19:57:45.187069 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.179007    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 22 19:57:45.187069 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.179010    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 22 19:57:45.187069 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.179012    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 22 19:57:45.187069 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.179015    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 22 19:57:45.187069 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.179017    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 22 19:57:45.187069 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.179020    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 22 19:57:45.187069 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.179022    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 22 19:57:45.187069 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.179025    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 22 19:57:45.187069 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.179027    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 22 19:57:45.187069 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.179030    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 22 19:57:45.187069 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.179032    2576 feature_gate.go:328] unrecognized feature gate: Example
Apr 22 19:57:45.187069 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.179035    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 22 19:57:45.187069 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.179038    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 22 19:57:45.187069 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.179041    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 22 19:57:45.187069 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.179043    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 22 19:57:45.187564 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.179045    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 22 19:57:45.187564 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.179831    2576 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Apr 22 19:57:45.187564 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.187545    2576 server.go:530] "Kubelet version" kubeletVersion="v1.33.9"
Apr 22 19:57:45.187564 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.187565    2576 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Apr 22 19:57:45.187751 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187639    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 22 19:57:45.187751 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187648    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 22 19:57:45.187751 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187653    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 22 19:57:45.187751 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187658    2576 feature_gate.go:328] unrecognized feature gate: Example
Apr 22 19:57:45.187751 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187663    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 22 19:57:45.187751 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187668    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 22 19:57:45.187751 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187673    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 22 19:57:45.187751 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187677    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 22 19:57:45.187751 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187682    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 22 19:57:45.187751 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187687    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 22 19:57:45.187751 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187691    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 22 19:57:45.187751 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187695    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 22 19:57:45.187751 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187700    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 22 19:57:45.187751 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187704    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 22 19:57:45.187751 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187707    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 22 19:57:45.187751 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187711    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 22 19:57:45.187751 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187715    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 22 19:57:45.187751 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187720    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 22 19:57:45.187751 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187724    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 22 19:57:45.187751 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187728    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 22 19:57:45.188702 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187732    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 22 19:57:45.188702 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187736    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 22 19:57:45.188702 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187740    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 22 19:57:45.188702 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187744    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 22 19:57:45.188702 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187748    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 22 19:57:45.188702 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187753    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 22 19:57:45.188702 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187757    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 22 19:57:45.188702 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187762    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 22 19:57:45.188702 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187766    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 22 19:57:45.188702 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187770    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 22 19:57:45.188702 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187774    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 22 19:57:45.188702 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187778    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 22 19:57:45.188702 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187783    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 22 19:57:45.188702 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187787    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 22 19:57:45.188702 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187791    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 22 19:57:45.188702 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187795    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 22 19:57:45.188702 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187799    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 22 19:57:45.188702 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187818    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 22 19:57:45.188702 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187822    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 22 19:57:45.188702 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187827    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 22 19:57:45.189241 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187831    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 22 19:57:45.189241 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187835    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 22 19:57:45.189241 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187838    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 22 19:57:45.189241 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187842    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 22 19:57:45.189241 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187846    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 22 19:57:45.189241 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187850    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 22 19:57:45.189241 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187854    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 22 19:57:45.189241 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187858    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 22 19:57:45.189241 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187862    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 22 19:57:45.189241 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187866    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 22 19:57:45.189241 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187870    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 22 19:57:45.189241 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187873    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 22 19:57:45.189241 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187877    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 22 19:57:45.189241 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187882    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 22 19:57:45.189241 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187886    2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 22 19:57:45.189241 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187890    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 22 19:57:45.189241 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187894    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 22 19:57:45.189241 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187898    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 22 19:57:45.189241 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187903    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 22 19:57:45.189241 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187908    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 22 19:57:45.189815 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187912    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 22 19:57:45.189815 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187916    2576 feature_gate.go:328] unrecognized feature gate: Example2
Apr 22 19:57:45.189815 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187920    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 22 19:57:45.189815 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187924    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 22 19:57:45.189815 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187928    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 22 19:57:45.189815 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187932    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 22 19:57:45.189815 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187936    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 22 19:57:45.189815 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187940    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 22 19:57:45.189815 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187946    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 22 19:57:45.189815 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187954    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 22 19:57:45.189815 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187959    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 22 19:57:45.189815 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187964    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 22 19:57:45.189815 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187968    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 22 19:57:45.189815 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187973    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 22 19:57:45.189815 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187977    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 22 19:57:45.189815 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187981    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 22 19:57:45.189815 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187986    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 22 19:57:45.189815 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187990    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 22 19:57:45.189815 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187993    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 22 19:57:45.190719 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.187998    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 22 19:57:45.190719 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188002    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 22 19:57:45.190719 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188006    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 22 19:57:45.190719 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188010    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 22 19:57:45.190719 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188014    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 22 19:57:45.190719 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188020    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 22 19:57:45.190719 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188026    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 22 19:57:45.190719 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.188034    2576 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Apr 22 19:57:45.190719 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188215    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 22 19:57:45.190719 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188224    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 22 19:57:45.190719 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188232    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 22 19:57:45.190719 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188239    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 22 19:57:45.190719 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188246    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 22 19:57:45.190719 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188251    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 22 19:57:45.190719 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188262    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 22 19:57:45.191372 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188267    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 22 19:57:45.191372 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188272    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 22 19:57:45.191372 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188276    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 22 19:57:45.191372 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188281    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 22 19:57:45.191372 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188286    2576 feature_gate.go:328] unrecognized feature gate: Example
Apr 22 19:57:45.191372 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188291    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 22 19:57:45.191372 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188295    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 22 19:57:45.191372 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188300    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 22 19:57:45.191372 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188304    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 22 19:57:45.191372 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188308    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 22 19:57:45.191372 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188312    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 22 19:57:45.191372 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188316    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 22 19:57:45.191372 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188320    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 22 19:57:45.191372 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188324    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 22 19:57:45.191372 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188328    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 22 19:57:45.191372 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188333    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 22 19:57:45.191372 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188337    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 22 19:57:45.191372 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188341    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 22 19:57:45.191372 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188345    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 22 19:57:45.191372 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188349    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 22 19:57:45.191853 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188353    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 22 19:57:45.191853 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188357    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 22 19:57:45.191853 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188361    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 22 19:57:45.191853 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188365    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 22 19:57:45.191853 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188369    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 22 19:57:45.191853 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188373    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 22 19:57:45.191853 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188377    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 22 19:57:45.191853 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188381    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 22 19:57:45.191853 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188385    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 22 19:57:45.191853 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188389    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 22 19:57:45.191853 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188394    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 22 19:57:45.191853 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188398    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 22 19:57:45.191853 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188402    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 22 19:57:45.191853 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188406    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 22 19:57:45.191853 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188410    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 22 19:57:45.191853 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188414    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 22 19:57:45.191853 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188418    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 22 19:57:45.191853 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188422    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 22 19:57:45.191853 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188426    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 22 19:57:45.191853 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188431    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 22 19:57:45.192480 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188435    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 22 19:57:45.192480 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188439    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 22 19:57:45.192480 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188443    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 22 19:57:45.192480 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188447    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 22 19:57:45.192480 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188451    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 22 19:57:45.192480 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188454    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 22 19:57:45.192480 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188458    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 22 19:57:45.192480 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188463    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 22 19:57:45.192480 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188467    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 22 19:57:45.192480 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188471    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 22 19:57:45.192480 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188475    2576 feature_gate.go:328] unrecognized feature gate: Example2
Apr 22 19:57:45.192480 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188479    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 22 19:57:45.192480 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188483    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 22 19:57:45.192480 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188487    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 22 19:57:45.192480 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188492    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 22 19:57:45.192480 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188498    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 22 19:57:45.192480 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188502    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 22 19:57:45.192480 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188506    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 22 19:57:45.192480 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188511    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 22 19:57:45.192943 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188514    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 22 19:57:45.192943 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188519    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 22 19:57:45.192943 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188522    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 22 19:57:45.192943 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188526    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 22 19:57:45.192943 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188531    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 22 19:57:45.192943 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188535    2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 22 19:57:45.192943 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188539    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 22 19:57:45.192943 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188543    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 22 19:57:45.192943 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188547    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 22 19:57:45.192943 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188551    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 22 19:57:45.192943 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188555    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 22 19:57:45.192943 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188559    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 22 19:57:45.192943 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188563    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 22 19:57:45.192943 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188568    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 22 19:57:45.192943 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188572    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 22 19:57:45.192943 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188576    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 22 19:57:45.192943 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188579    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 22 19:57:45.192943 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188583    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 22 19:57:45.192943 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188587    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 22 19:57:45.192943 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:45.188591    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 22 19:57:45.193432 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.188599    2576 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Apr 22 19:57:45.193432 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.189342    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Apr 22 19:57:45.193432 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.191927    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Apr 22 19:57:45.194223 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.194184    2576 server.go:1019] "Starting client certificate rotation"
Apr 22 19:57:45.194333 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.194286    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Apr 22 19:57:45.194333 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.194319    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Apr 22 19:57:45.215540 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.215521    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Apr 22 19:57:45.219425 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.219410    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Apr 22 19:57:45.233211 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.233192    2576 log.go:25] "Validated CRI v1 runtime API"
Apr 22 19:57:45.238057 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.238042    2576 log.go:25] "Validated CRI v1 image API"
Apr 22 19:57:45.239347 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.239318    2576 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Apr 22 19:57:45.242924 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.242889    2576 fs.go:135] Filesystem UUIDs: map[749b545d-a158-45d6-a17a-474266eb58af:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c2fd54ac-2d0d-41d5-bd7e-5910ccf1d6fa:/dev/nvme0n1p3]
Apr 22 19:57:45.243012 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.242921    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Apr 22 19:57:45.245584 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.245563    2576 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Apr 22 19:57:45.248220 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.248095    2576 manager.go:217] Machine: {Timestamp:2026-04-22 19:57:45.247019702 +0000 UTC m=+0.346975959 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099847 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29c36b6d5486c020fce7ae1bdbef60 SystemUUID:ec29c36b-6d54-86c0-20fc-e7ae1bdbef60 BootID:97a3a66c-7b5d-4fe1-8092-26ab10caa733 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3f:e6:5d:1f:71 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3f:e6:5d:1f:71 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:fa:e3:79:19:6d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Apr 22 19:57:45.248220 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.248216    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Apr 22 19:57:45.248323 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.248294    2576 manager.go:233] Version: {KernelVersion:5.14.0-570.107.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260414-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Apr 22 19:57:45.250503 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.250482    2576 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Apr 22 19:57:45.250656 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.250506    2576 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-245.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Apr 22 19:57:45.250702 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.250666    2576 topology_manager.go:138] "Creating topology manager with none policy"
Apr 22 19:57:45.250702 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.250674    2576 container_manager_linux.go:306] "Creating device plugin manager"
Apr 22 19:57:45.250702 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.250687    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Apr 22 19:57:45.250702 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.250701    2576 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Apr 22 19:57:45.251398 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.251388    2576 state_mem.go:36] "Initialized new in-memory state store"
Apr 22 19:57:45.251504 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.251495    2576 server.go:1267] "Using root directory" path="/var/lib/kubelet"
Apr 22 19:57:45.253445 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.253435    2576 kubelet.go:491] "Attempting to sync node with API server"
Apr 22 19:57:45.253481 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.253455    2576 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
Apr 22 19:57:45.254186 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.254176    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Apr 22 19:57:45.254230 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.254189    2576 kubelet.go:397] "Adding apiserver pod source"
Apr 22 19:57:45.254230 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.254199    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Apr 22 19:57:45.255092 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.255081    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Apr 22 19:57:45.255136 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.255098    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Apr 22 19:57:45.257685 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.257668    2576 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.10-2.rhaos4.20.gita4d0894.el9" apiVersion="v1"
Apr 22 19:57:45.258837 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.258824    2576 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Apr 22 19:57:45.260462 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.260450    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Apr 22 19:57:45.260522 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.260467    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Apr 22 19:57:45.260522 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.260474    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Apr 22 19:57:45.260522 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.260480    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Apr 22 19:57:45.260522 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.260485    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Apr 22 19:57:45.260522 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.260492    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Apr 22 19:57:45.260522 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.260499    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Apr 22 19:57:45.260522 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.260504    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Apr 22 19:57:45.260522 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.260511    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Apr 22 19:57:45.260522 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.260517    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Apr 22 19:57:45.260522 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.260525    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Apr 22 19:57:45.260837 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.260535    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Apr 22 19:57:45.260837 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.260560    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Apr 22 19:57:45.260837 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.260565    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
Apr 22 19:57:45.264091 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.264076    2576 watchdog_linux.go:99] "Systemd watchdog is not enabled"
Apr 22 19:57:45.264194 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.264121    2576 server.go:1295] "Started kubelet"
Apr 22 19:57:45.264267 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.264224    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Apr 22 19:57:45.267572 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.267522    2576 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Apr 22 19:57:45.267662 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.267601    2576 server_v1.go:47] "podresources" method="list" useActivePods=true
Apr 22 19:57:45.267849 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.267822    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9kf4g"
Apr 22 19:57:45.268410 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.268389    2576 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-245.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Apr 22 19:57:45.268619 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.268597    2576 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Apr 22 19:57:45.268655 ip-10-0-135-245 systemd[1]: Started Kubernetes Kubelet.
Apr 22 19:57:45.269418 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:45.269393    2576 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Apr 22 19:57:45.269501 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:45.269400    2576 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-245.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Apr 22 19:57:45.271775 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.271760    2576 server.go:317] "Adding debug handlers to kubelet server"
Apr 22 19:57:45.272841 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:45.271583    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-245.ec2.internal.18a8c6195ec4acd2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-245.ec2.internal,UID:ip-10-0-135-245.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-245.ec2.internal,},FirstTimestamp:2026-04-22 19:57:45.264090322 +0000 UTC m=+0.364046583,LastTimestamp:2026-04-22 19:57:45.264090322 +0000 UTC m=+0.364046583,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-245.ec2.internal,}"
Apr 22 19:57:45.272841 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.272834    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9kf4g"
Apr 22 19:57:45.276125 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.276107    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Apr 22 19:57:45.276569 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.276553    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Apr 22 19:57:45.276871 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:45.276826    2576 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Apr 22 19:57:45.277088 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.277077    2576 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
Apr 22 19:57:45.277164 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.277078    2576 volume_manager.go:295] "The desired_state_of_world populator starts"
Apr 22 19:57:45.277164 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.277099    2576 volume_manager.go:297] "Starting Kubelet Volume Manager"
Apr 22 19:57:45.277247 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.277191    2576 reconstruct.go:97] "Volume reconstruction finished"
Apr 22 19:57:45.277247 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.277198    2576 reconciler.go:26] "Reconciler: start to sync state"
Apr 22 19:57:45.277324 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:45.277261    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-245.ec2.internal\" not found"
Apr 22 19:57:45.277560 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.277547    2576 factory.go:153] Registering CRI-O factory
Apr 22 19:57:45.277612 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.277568    2576 factory.go:223] Registration of the crio container factory successfully
Apr 22 19:57:45.277612 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.277609    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Apr 22 19:57:45.277671 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.277616    2576 factory.go:55] Registering systemd factory
Apr 22 19:57:45.277671 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.277621    2576 factory.go:223] Registration of the systemd container factory successfully
Apr 22 19:57:45.277671 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.277637    2576 factory.go:103] Registering Raw factory
Apr 22 19:57:45.277671 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.277646    2576 manager.go:1196] Started watching for new ooms in manager
Apr 22 19:57:45.278065 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.278054    2576 manager.go:319] Starting recovery of all containers
Apr 22 19:57:45.278449 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:45.278428    2576 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Apr 22 19:57:45.278550 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:45.278532    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-245.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Apr 22 19:57:45.290788 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.290646    2576 manager.go:324] Recovery completed
Apr 22 19:57:45.292342 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:45.292321    2576 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Apr 22 19:57:45.295132 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.295121    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 22 19:57:45.297559 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.297533    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-245.ec2.internal" event="NodeHasSufficientMemory"
Apr 22 19:57:45.297625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.297565    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-245.ec2.internal" event="NodeHasNoDiskPressure"
Apr 22 19:57:45.297625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.297578    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-245.ec2.internal" event="NodeHasSufficientPID"
Apr 22 19:57:45.298527 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.298508    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Apr 22 19:57:45.298600 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.298527    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Apr 22 19:57:45.298600 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.298550    2576 state_mem.go:36] "Initialized new in-memory state store"
Apr 22 19:57:45.301259 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.301240    2576 policy_none.go:49] "None policy: Start"
Apr 22 19:57:45.301337 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.301275    2576 memory_manager.go:186] "Starting memorymanager" policy="None"
Apr 22 19:57:45.301337 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.301288    2576 state_mem.go:35] "Initializing new in-memory state store"
Apr 22 19:57:45.338767 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.338753    2576 manager.go:341] "Starting Device Plugin manager"
Apr 22 19:57:45.344015 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:45.338782    2576 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Apr 22 19:57:45.344015 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.338797    2576 server.go:85] "Starting device plugin registration server"
Apr 22 19:57:45.344015 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.338992    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Apr 22 19:57:45.344015 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.339002    2576 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Apr 22 19:57:45.344015 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.339109    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Apr 22 19:57:45.344015 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.339204    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Apr 22 19:57:45.344015 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.339210    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Apr 22 19:57:45.344015 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:45.339908    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Apr 22 19:57:45.344015 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:45.339953    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-245.ec2.internal\" not found"
Apr 22 19:57:45.367527 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.367500    2576 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
Apr 22 19:57:45.368904 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.368888    2576 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
Apr 22 19:57:45.368956 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.368917    2576 status_manager.go:230] "Starting to sync pod status with apiserver"
Apr 22 19:57:45.368956 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.368939    2576 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Apr 22 19:57:45.368956 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.368948    2576 kubelet.go:2451] "Starting kubelet main sync loop"
Apr 22 19:57:45.369075 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:45.368987    2576 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Apr 22 19:57:45.371322 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.371302    2576 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 22 19:57:45.439552 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.439479    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 22 19:57:45.440607 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.440587    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-245.ec2.internal" event="NodeHasSufficientMemory"
Apr 22 19:57:45.440714 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.440620    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-245.ec2.internal" event="NodeHasNoDiskPressure"
Apr 22 19:57:45.440714 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.440634    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-245.ec2.internal" event="NodeHasSufficientPID"
Apr 22 19:57:45.440714 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.440664    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-245.ec2.internal"
Apr 22 19:57:45.448177 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.448160    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-245.ec2.internal"
Apr 22 19:57:45.448268 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:45.448186    2576 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-245.ec2.internal\": node \"ip-10-0-135-245.ec2.internal\" not found"
Apr 22 19:57:45.469592 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.469565    2576 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-245.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-245.ec2.internal"]
Apr 22 19:57:45.469697 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.469636    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 22 19:57:45.472279 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:45.472262    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-245.ec2.internal\" not found"
Apr 22 19:57:45.476969 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.476950    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-245.ec2.internal" event="NodeHasSufficientMemory"
Apr 22 19:57:45.477044 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.476984    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-245.ec2.internal" event="NodeHasNoDiskPressure"
Apr 22 19:57:45.477044 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.476995    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-245.ec2.internal" event="NodeHasSufficientPID"
Apr 22 19:57:45.478120 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.478108    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 22 19:57:45.478261 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.478245    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-245.ec2.internal"
Apr 22 19:57:45.478300 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.478275    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 22 19:57:45.478899 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.478881    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-245.ec2.internal" event="NodeHasSufficientMemory"
Apr 22 19:57:45.478967 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.478908    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-245.ec2.internal" event="NodeHasNoDiskPressure"
Apr 22 19:57:45.478967 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.478882    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-245.ec2.internal" event="NodeHasSufficientMemory"
Apr 22 19:57:45.478967 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.478946    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-245.ec2.internal" event="NodeHasNoDiskPressure"
Apr 22 19:57:45.478967 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.478959    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-245.ec2.internal" event="NodeHasSufficientPID"
Apr 22 19:57:45.479102 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.478919    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-245.ec2.internal" event="NodeHasSufficientPID"
Apr 22 19:57:45.479948 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.479922    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-245.ec2.internal"
Apr 22 19:57:45.480048 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.479954    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 22 19:57:45.480718 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.480704    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-245.ec2.internal" event="NodeHasSufficientMemory"
Apr 22 19:57:45.480796 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.480732    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-245.ec2.internal" event="NodeHasNoDiskPressure"
Apr 22 19:57:45.480796 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.480745    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-245.ec2.internal" event="NodeHasSufficientPID"
Apr 22 19:57:45.503740 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:45.503715    2576 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-245.ec2.internal\" not found" node="ip-10-0-135-245.ec2.internal"
Apr 22 19:57:45.508538 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:45.508523    2576 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-245.ec2.internal\" not found" node="ip-10-0-135-245.ec2.internal"
Apr 22 19:57:45.572541 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:45.572509    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-245.ec2.internal\" not found"
Apr 22 19:57:45.577761 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.577741    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0353ff154730f2cf2a2ea68e586089e5-config\") pod \"kube-apiserver-proxy-ip-10-0-135-245.ec2.internal\" (UID: \"0353ff154730f2cf2a2ea68e586089e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-245.ec2.internal"
Apr 22 19:57:45.672999 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:45.672960    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-245.ec2.internal\" not found"
Apr 22 19:57:45.678275 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.678253    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0353ff154730f2cf2a2ea68e586089e5-config\") pod \"kube-apiserver-proxy-ip-10-0-135-245.ec2.internal\" (UID: \"0353ff154730f2cf2a2ea68e586089e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-245.ec2.internal"
Apr 22 19:57:45.678355 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.678281    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7eff41dfcc7a8a8786c50b0bcb949e70-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-245.ec2.internal\" (UID: \"7eff41dfcc7a8a8786c50b0bcb949e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-245.ec2.internal"
Apr 22 19:57:45.678355 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.678300    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eff41dfcc7a8a8786c50b0bcb949e70-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-245.ec2.internal\" (UID: \"7eff41dfcc7a8a8786c50b0bcb949e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-245.ec2.internal"
Apr 22 19:57:45.678427 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.678353    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0353ff154730f2cf2a2ea68e586089e5-config\") pod \"kube-apiserver-proxy-ip-10-0-135-245.ec2.internal\" (UID: \"0353ff154730f2cf2a2ea68e586089e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-245.ec2.internal"
Apr 22 19:57:45.773722 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:45.773660    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-245.ec2.internal\" not found"
Apr 22 19:57:45.778938 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.778918    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7eff41dfcc7a8a8786c50b0bcb949e70-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-245.ec2.internal\" (UID: \"7eff41dfcc7a8a8786c50b0bcb949e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-245.ec2.internal"
Apr 22 19:57:45.779009 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.778953    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eff41dfcc7a8a8786c50b0bcb949e70-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-245.ec2.internal\" (UID: \"7eff41dfcc7a8a8786c50b0bcb949e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-245.ec2.internal"
Apr 22 19:57:45.779009 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.778989    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eff41dfcc7a8a8786c50b0bcb949e70-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-245.ec2.internal\" (UID: \"7eff41dfcc7a8a8786c50b0bcb949e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-245.ec2.internal"
Apr 22 19:57:45.779009 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.779003    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7eff41dfcc7a8a8786c50b0bcb949e70-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-245.ec2.internal\" (UID: \"7eff41dfcc7a8a8786c50b0bcb949e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-245.ec2.internal"
Apr 22 19:57:45.805089 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.805065    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-245.ec2.internal"
Apr 22 19:57:45.810390 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:45.810374    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-245.ec2.internal"
Apr 22 19:57:45.874481 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:45.874434    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-245.ec2.internal\" not found"
Apr 22 19:57:45.975003 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:45.974959    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-245.ec2.internal\" not found"
Apr 22 19:57:46.075467 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:46.075391    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-245.ec2.internal\" not found"
Apr 22 19:57:46.099238 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.099214    2576 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 22 19:57:46.176870 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.176849    2576 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-245.ec2.internal"
Apr 22 19:57:46.190443 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.190413    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Apr 22 19:57:46.192500 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.192480    2576 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-245.ec2.internal"
Apr 22 19:57:46.192591 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.192522    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Apr 22 19:57:46.192719 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.192697    2576 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Apr 22 19:57:46.192776 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.192717    2576 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Apr 22 19:57:46.215283 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.215262    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Apr 22 19:57:46.218237 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.218220    2576 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 22 19:57:46.255074 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.255048    2576 apiserver.go:52] "Watching apiserver"
Apr 22 19:57:46.265219 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.265200    2576 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Apr 22 19:57:46.266268 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.266248    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-pq8s7","openshift-network-operator/iptables-alerter-fd2xn","openshift-ovn-kubernetes/ovnkube-node-rbkp9","openshift-image-registry/node-ca-tn59d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-245.ec2.internal","openshift-multus/multus-additional-cni-plugins-q25vj","openshift-multus/multus-rzjvv","openshift-multus/network-metrics-daemon-lkqdc","kube-system/konnectivity-agent-pmrkj","kube-system/kube-apiserver-proxy-ip-10-0-135-245.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh","openshift-cluster-node-tuning-operator/tuned-vzjns","openshift-dns/node-resolver-rmblf"]
Apr 22 19:57:46.272233 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.272212    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.274285 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.274263    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fd2xn"
Apr 22 19:57:46.274404 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.274386    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Apr 22 19:57:46.274464 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.274422    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-04-21 19:52:45 +0000 UTC" deadline="2028-01-11 00:17:50.124070792 +0000 UTC"
Apr 22 19:57:46.274464 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.274448    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15076h20m3.849626303s"
Apr 22 19:57:46.274567 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.274393    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.274692 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.274614    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Apr 22 19:57:46.274865 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.274842    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Apr 22 19:57:46.275134 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.275117    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f6lmc\""
Apr 22 19:57:46.275234 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.275123    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Apr 22 19:57:46.276247 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.276232    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Apr 22 19:57:46.276351 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.276253    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Apr 22 19:57:46.276839 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.276820    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tn59d"
Apr 22 19:57:46.278445 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.278067    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2h9bq\""
Apr 22 19:57:46.278445 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.278180    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Apr 22 19:57:46.278445 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.278204    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Apr 22 19:57:46.278445 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.278254    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Apr 22 19:57:46.278445 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.278280    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tsf74\""
Apr 22 19:57:46.278445 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.278343    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Apr 22 19:57:46.278445 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.278390    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Apr 22 19:57:46.278786 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.278500    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Apr 22 19:57:46.278786 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.278539    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Apr 22 19:57:46.278786 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.278559    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Apr 22 19:57:46.278927 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.278845    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Apr 22 19:57:46.278927 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.278859    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Apr 22 19:57:46.278990 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.278942    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Apr 22 19:57:46.279047 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.279034    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-txvmq\""
Apr 22 19:57:46.279248 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.279233    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.281319 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.281302    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Apr 22 19:57:46.281450 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.281435    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:57:46.281566 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:46.281542    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pq8s7" podUID="7f4a64a7-a0cc-426a-9469-5cb2654b3b1d"
Apr 22 19:57:46.281612 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.281601    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lkmrr\""
Apr 22 19:57:46.281710 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.281696    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Apr 22 19:57:46.281808 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.281795    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-log-socket\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.281850 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.281827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-ovn-node-metrics-cert\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.281850 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.281843    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-ovnkube-script-lib\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.281935 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.281858    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blg28\" (UniqueName: \"kubernetes.io/projected/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-kube-api-access-blg28\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.281935 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.281872    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-cnibin\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.281935 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.281896    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-hostroot\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.282045 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.281968    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmmj6\" (UniqueName: \"kubernetes.io/projected/d5f06d35-71cb-4bf8-af1b-e806af89003e-kube-api-access-tmmj6\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.282045 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282000    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc0bc97c-7cea-48d0-a254-fb23e782e797-cnibin\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.282045 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282029    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc0bc97c-7cea-48d0-a254-fb23e782e797-cni-binary-copy\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.282162 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282056    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-multus-conf-dir\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.282162 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282079    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-etc-openvswitch\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.282162 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282105    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.282162 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282130    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/671f3faa-9804-447f-acb1-14a0c42815dd-host\") pod \"node-ca-tn59d\" (UID: \"671f3faa-9804-447f-acb1-14a0c42815dd\") " pod="openshift-image-registry/node-ca-tn59d"
Apr 22 19:57:46.282314 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282172    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc0bc97c-7cea-48d0-a254-fb23e782e797-system-cni-dir\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.282314 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc0bc97c-7cea-48d0-a254-fb23e782e797-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.282314 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282218    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5f06d35-71cb-4bf8-af1b-e806af89003e-cni-binary-copy\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.282314 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282242    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-multus-socket-dir-parent\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.282314 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282268    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-host-var-lib-kubelet\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.282314 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5f06d35-71cb-4bf8-af1b-e806af89003e-multus-daemon-config\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.282535 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282350    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ddb6ca29-0d32-41ce-a7d6-8e737ff47930-host-slash\") pod \"iptables-alerter-fd2xn\" (UID: \"ddb6ca29-0d32-41ce-a7d6-8e737ff47930\") " pod="openshift-network-operator/iptables-alerter-fd2xn"
Apr 22 19:57:46.282535 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282403    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsj7w\" (UniqueName: \"kubernetes.io/projected/ddb6ca29-0d32-41ce-a7d6-8e737ff47930-kube-api-access-jsj7w\") pod \"iptables-alerter-fd2xn\" (UID: \"ddb6ca29-0d32-41ce-a7d6-8e737ff47930\") " pod="openshift-network-operator/iptables-alerter-fd2xn"
Apr 22 19:57:46.282535 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282429    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dc0bc97c-7cea-48d0-a254-fb23e782e797-os-release\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.282535 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282448    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdkcs\" (UniqueName: \"kubernetes.io/projected/dc0bc97c-7cea-48d0-a254-fb23e782e797-kube-api-access-cdkcs\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.282535 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282489    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-host-kubelet\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.282535 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282526    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-systemd-units\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.282727 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282563    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-run-systemd\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.282727 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282598    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-var-lib-openvswitch\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.282727 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282645    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc0bc97c-7cea-48d0-a254-fb23e782e797-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.282727 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282672    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc0bc97c-7cea-48d0-a254-fb23e782e797-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.282727 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282696    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-etc-kubernetes\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.282727 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282717    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ddb6ca29-0d32-41ce-a7d6-8e737ff47930-iptables-alerter-script\") pod \"iptables-alerter-fd2xn\" (UID: \"ddb6ca29-0d32-41ce-a7d6-8e737ff47930\") " pod="openshift-network-operator/iptables-alerter-fd2xn"
Apr 22 19:57:46.282980 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282741    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-run-ovn\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.282980 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282765    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-node-log\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.282980 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282787    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-host-cni-bin\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.282980 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282809    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/671f3faa-9804-447f-acb1-14a0c42815dd-serviceca\") pod \"node-ca-tn59d\" (UID: \"671f3faa-9804-447f-acb1-14a0c42815dd\") " pod="openshift-image-registry/node-ca-tn59d"
Apr 22 19:57:46.282980 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282835    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-os-release\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.282980 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282863    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-host-run-k8s-cni-cncf-io\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.282980 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282886    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-host-run-netns\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.282980 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-host-run-ovn-kubernetes\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.282980 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282948    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8r68\" (UniqueName: \"kubernetes.io/projected/671f3faa-9804-447f-acb1-14a0c42815dd-kube-api-access-c8r68\") pod \"node-ca-tn59d\" (UID: \"671f3faa-9804-447f-acb1-14a0c42815dd\") " pod="openshift-image-registry/node-ca-tn59d"
Apr 22 19:57:46.283277 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.282973    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-system-cni-dir\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.283277 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.283005    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-host-run-netns\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.283277 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.283025    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-host-var-lib-cni-bin\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.283277 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.283042    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-host-var-lib-cni-multus\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.283277 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.283076    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-host-slash\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.283277 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.283102    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-host-cni-netd\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.283277 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.283122    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-ovnkube-config\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.283277 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.283162    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-env-overrides\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.283277 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.283193    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-multus-cni-dir\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.283277 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.283207    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-host-run-multus-certs\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.283277 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.283222    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-run-openvswitch\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.283821 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.283811    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:57:46.284116 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:46.284089    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkqdc" podUID="7dd397be-1edf-42eb-8b45-a3291c8c3b7f"
Apr 22 19:57:46.286844 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:46.286812    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0353ff154730f2cf2a2ea68e586089e5.slice/crio-ef327383e391cd0dde9a251741e1c31aa1fa1cced1781cc3d8d6c6a000d214ac WatchSource:0}: Error finding container ef327383e391cd0dde9a251741e1c31aa1fa1cced1781cc3d8d6c6a000d214ac: Status 404 returned error can't find the container with id ef327383e391cd0dde9a251741e1c31aa1fa1cced1781cc3d8d6c6a000d214ac
Apr 22 19:57:46.287204 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.287182    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pmrkj"
Apr 22 19:57:46.289364 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.289342    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.289817 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.289803    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-x4jwj\""
Apr 22 19:57:46.291733 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.291711    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.292081 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.292061    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Apr 22 19:57:46.293290 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.293272    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 22 19:57:46.294294 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.294278    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rmblf"
Apr 22 19:57:46.294375 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.294303    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Apr 22 19:57:46.294654 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.294632    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Apr 22 19:57:46.294870 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.294848    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Apr 22 19:57:46.294870 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.294859    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Apr 22 19:57:46.295104 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.295086    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4rlrc\""
Apr 22 19:57:46.295191 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.295116    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Apr 22 19:57:46.295243 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.295209    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-88jdn\""
Apr 22 19:57:46.295243 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.295227    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Apr 22 19:57:46.296941 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.296927    2576 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Apr 22 19:57:46.297496 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:46.297472    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eff41dfcc7a8a8786c50b0bcb949e70.slice/crio-9743b3b015b5fe1ad40266a5f201e7af4932458f0dae9cca0256682f7cddc876 WatchSource:0}: Error finding container 9743b3b015b5fe1ad40266a5f201e7af4932458f0dae9cca0256682f7cddc876: Status 404 returned error can't find the container with id 9743b3b015b5fe1ad40266a5f201e7af4932458f0dae9cca0256682f7cddc876
Apr 22 19:57:46.297906 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.297721    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Apr 22 19:57:46.297906 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.297747    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Apr 22 19:57:46.297906 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.297866    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4fjn6\""
Apr 22 19:57:46.371810 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.371764    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-245.ec2.internal" event={"ID":"7eff41dfcc7a8a8786c50b0bcb949e70","Type":"ContainerStarted","Data":"9743b3b015b5fe1ad40266a5f201e7af4932458f0dae9cca0256682f7cddc876"}
Apr 22 19:57:46.372599 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.372578    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-245.ec2.internal" event={"ID":"0353ff154730f2cf2a2ea68e586089e5","Type":"ContainerStarted","Data":"ef327383e391cd0dde9a251741e1c31aa1fa1cced1781cc3d8d6c6a000d214ac"}
Apr 22 19:57:46.378186 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.378169    2576 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 22 19:57:46.378339 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.378325    2576 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
Apr 22 19:57:46.384272 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384247    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-etc-openvswitch\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.384367 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384278    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/671f3faa-9804-447f-acb1-14a0c42815dd-host\") pod \"node-ca-tn59d\" (UID: \"671f3faa-9804-447f-acb1-14a0c42815dd\") " pod="openshift-image-registry/node-ca-tn59d"
Apr 22 19:57:46.384367 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384307    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-etc-sysconfig\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.384447 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384362    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-etc-openvswitch\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.384447 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384361    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/671f3faa-9804-447f-acb1-14a0c42815dd-host\") pod \"node-ca-tn59d\" (UID: \"671f3faa-9804-447f-acb1-14a0c42815dd\") " pod="openshift-image-registry/node-ca-tn59d"
Apr 22 19:57:46.384447 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384409    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-etc-tuned\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.384447 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384429    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs\") pod \"network-metrics-daemon-lkqdc\" (UID: \"7dd397be-1edf-42eb-8b45-a3291c8c3b7f\") " pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:57:46.384447 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384445    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d5d3cbed-7c79-4ea8-a02c-2992c80af473-konnectivity-ca\") pod \"konnectivity-agent-pmrkj\" (UID: \"d5d3cbed-7c79-4ea8-a02c-2992c80af473\") " pod="kube-system/konnectivity-agent-pmrkj"
Apr 22 19:57:46.384714 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384461    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34b46484-7515-431f-9460-068a93fec7e4-device-dir\") pod \"aws-ebs-csi-driver-node-65cmh\" (UID: \"34b46484-7515-431f-9460-068a93fec7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.384714 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384479    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc0bc97c-7cea-48d0-a254-fb23e782e797-system-cni-dir\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.384714 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384500    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5f06d35-71cb-4bf8-af1b-e806af89003e-cni-binary-copy\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.384714 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384530    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-multus-socket-dir-parent\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.384714 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384555    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5f06d35-71cb-4bf8-af1b-e806af89003e-multus-daemon-config\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.384714 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384609    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc0bc97c-7cea-48d0-a254-fb23e782e797-system-cni-dir\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.384714 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384619    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-multus-socket-dir-parent\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.385036 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384724    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ddb6ca29-0d32-41ce-a7d6-8e737ff47930-host-slash\") pod \"iptables-alerter-fd2xn\" (UID: \"ddb6ca29-0d32-41ce-a7d6-8e737ff47930\") " pod="openshift-network-operator/iptables-alerter-fd2xn"
Apr 22 19:57:46.385036 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384749    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jsj7w\" (UniqueName: \"kubernetes.io/projected/ddb6ca29-0d32-41ce-a7d6-8e737ff47930-kube-api-access-jsj7w\") pod \"iptables-alerter-fd2xn\" (UID: \"ddb6ca29-0d32-41ce-a7d6-8e737ff47930\") " pod="openshift-network-operator/iptables-alerter-fd2xn"
Apr 22 19:57:46.385036 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384765    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-run-systemd\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.385036 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384781    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c8r68\" (UniqueName: \"kubernetes.io/projected/671f3faa-9804-447f-acb1-14a0c42815dd-kube-api-access-c8r68\") pod \"node-ca-tn59d\" (UID: \"671f3faa-9804-447f-acb1-14a0c42815dd\") " pod="openshift-image-registry/node-ca-tn59d"
Apr 22 19:57:46.385036 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384806    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-etc-modprobe-d\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.385036 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384832    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/671f3faa-9804-447f-acb1-14a0c42815dd-serviceca\") pod \"node-ca-tn59d\" (UID: \"671f3faa-9804-447f-acb1-14a0c42815dd\") " pod="openshift-image-registry/node-ca-tn59d"
Apr 22 19:57:46.385036 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384843    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ddb6ca29-0d32-41ce-a7d6-8e737ff47930-host-slash\") pod \"iptables-alerter-fd2xn\" (UID: \"ddb6ca29-0d32-41ce-a7d6-8e737ff47930\") " pod="openshift-network-operator/iptables-alerter-fd2xn"
Apr 22 19:57:46.385036 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384852    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-run-systemd\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.385036 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384860    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj7k8\" (UniqueName: \"kubernetes.io/projected/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-kube-api-access-rj7k8\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.385036 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.384977    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ch42\" (UniqueName: \"kubernetes.io/projected/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-kube-api-access-9ch42\") pod \"network-metrics-daemon-lkqdc\" (UID: \"7dd397be-1edf-42eb-8b45-a3291c8c3b7f\") " pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:57:46.385036 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385006    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5f06d35-71cb-4bf8-af1b-e806af89003e-cni-binary-copy\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.385536 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385008    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34b46484-7515-431f-9460-068a93fec7e4-sys-fs\") pod \"aws-ebs-csi-driver-node-65cmh\" (UID: \"34b46484-7515-431f-9460-068a93fec7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.385536 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385065    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc0bc97c-7cea-48d0-a254-fb23e782e797-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.385536 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385100    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc0bc97c-7cea-48d0-a254-fb23e782e797-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.385536 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385113    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5f06d35-71cb-4bf8-af1b-e806af89003e-multus-daemon-config\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.385536 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385124    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-run-ovn\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.385536 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385163    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-host-cni-bin\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.385536 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385191    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-etc-sysctl-conf\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.385536 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385195    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-run-ovn\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.385536 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385214    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-os-release\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.385536 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385244    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/671f3faa-9804-447f-acb1-14a0c42815dd-serviceca\") pod \"node-ca-tn59d\" (UID: \"671f3faa-9804-447f-acb1-14a0c42815dd\") " pod="openshift-image-registry/node-ca-tn59d"
Apr 22 19:57:46.385536 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385241    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-host-cni-bin\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.385536 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385248    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-host\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.385536 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385276    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-os-release\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.385536 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385289    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f12eca63-f2df-4781-97dd-7968beea64ba-hosts-file\") pod \"node-resolver-rmblf\" (UID: \"f12eca63-f2df-4781-97dd-7968beea64ba\") " pod="openshift-dns/node-resolver-rmblf"
Apr 22 19:57:46.385536 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385307    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34b46484-7515-431f-9460-068a93fec7e4-socket-dir\") pod \"aws-ebs-csi-driver-node-65cmh\" (UID: \"34b46484-7515-431f-9460-068a93fec7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.385536 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385334    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-system-cni-dir\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.385536 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385349    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-ovnkube-config\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.386220 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385364    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-env-overrides\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.386220 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385379    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-multus-cni-dir\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.386220 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385395    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-run-openvswitch\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.386220 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385418    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-log-socket\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.386220 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385429    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-system-cni-dir\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.386220 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385435    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f12eca63-f2df-4781-97dd-7968beea64ba-tmp-dir\") pod \"node-resolver-rmblf\" (UID: \"f12eca63-f2df-4781-97dd-7968beea64ba\") " pod="openshift-dns/node-resolver-rmblf"
Apr 22 19:57:46.386220 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385456    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-multus-cni-dir\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.386220 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385467    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-run-openvswitch\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.386220 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385484    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34b46484-7515-431f-9460-068a93fec7e4-registration-dir\") pod \"aws-ebs-csi-driver-node-65cmh\" (UID: \"34b46484-7515-431f-9460-068a93fec7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.386220 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385501    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-log-socket\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.386220 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385517    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-cnibin\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.386220 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385555    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-hostroot\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.386220 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385582    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tmmj6\" (UniqueName: \"kubernetes.io/projected/d5f06d35-71cb-4bf8-af1b-e806af89003e-kube-api-access-tmmj6\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.386220 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385602    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc0bc97c-7cea-48d0-a254-fb23e782e797-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.386220 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385609    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-cnibin\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.386220 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385607    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc0bc97c-7cea-48d0-a254-fb23e782e797-cnibin\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.386220 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385625    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-hostroot\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.386220 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385643    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc0bc97c-7cea-48d0-a254-fb23e782e797-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.386750 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385651    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc0bc97c-7cea-48d0-a254-fb23e782e797-cnibin\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.386750 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385720    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc0bc97c-7cea-48d0-a254-fb23e782e797-cni-binary-copy\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.386750 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385740    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-multus-conf-dir\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.386750 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385757    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.386750 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385781    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-run\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.386750 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385800    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-multus-conf-dir\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.386750 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385806    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8vfm\" (UniqueName: \"kubernetes.io/projected/34b46484-7515-431f-9460-068a93fec7e4-kube-api-access-j8vfm\") pod \"aws-ebs-csi-driver-node-65cmh\" (UID: \"34b46484-7515-431f-9460-068a93fec7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.386750 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385822    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.386750 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385833    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc0bc97c-7cea-48d0-a254-fb23e782e797-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.386750 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385849    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-host-var-lib-kubelet\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.386750 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385854    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-env-overrides\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.386750 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385867    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-var-lib-openvswitch\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.386750 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385891    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tfnx\" (UniqueName: \"kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx\") pod \"network-check-target-pq8s7\" (UID: \"7f4a64a7-a0cc-426a-9469-5cb2654b3b1d\") " pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:57:46.386750 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385891    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-host-var-lib-kubelet\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.386750 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385959    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-var-lib-openvswitch\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.386750 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385994    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-ovnkube-config\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.386750 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385997    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc0bc97c-7cea-48d0-a254-fb23e782e797-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.387270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.385972    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dc0bc97c-7cea-48d0-a254-fb23e782e797-os-release\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.387270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386037    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cdkcs\" (UniqueName: \"kubernetes.io/projected/dc0bc97c-7cea-48d0-a254-fb23e782e797-kube-api-access-cdkcs\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.387270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386057    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dc0bc97c-7cea-48d0-a254-fb23e782e797-os-release\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.387270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386063    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-host-kubelet\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.387270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386116    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-systemd-units\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.387270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386142    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-systemd-units\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.387270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386174    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc0bc97c-7cea-48d0-a254-fb23e782e797-cni-binary-copy\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.387270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386190    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-node-log\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.387270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386196    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-host-kubelet\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.387270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386220    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-node-log\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.387270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386222    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-sys\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.387270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386250    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-etc-kubernetes\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.387270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386265    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ddb6ca29-0d32-41ce-a7d6-8e737ff47930-iptables-alerter-script\") pod \"iptables-alerter-fd2xn\" (UID: \"ddb6ca29-0d32-41ce-a7d6-8e737ff47930\") " pod="openshift-network-operator/iptables-alerter-fd2xn"
Apr 22 19:57:46.387270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386294    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-etc-kubernetes\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.387270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386292    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-etc-kubernetes\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.387270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386323    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34b46484-7515-431f-9460-068a93fec7e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-65cmh\" (UID: \"34b46484-7515-431f-9460-068a93fec7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.387270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386338    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34b46484-7515-431f-9460-068a93fec7e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-65cmh\" (UID: \"34b46484-7515-431f-9460-068a93fec7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.387763 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386358    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-host-run-k8s-cni-cncf-io\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.387763 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386373    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-host-run-netns\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.387763 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386395    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-host-run-ovn-kubernetes\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.387763 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386410    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-host-run-netns\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.387763 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386415    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-host-run-k8s-cni-cncf-io\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.387763 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386425    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-host-var-lib-cni-bin\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.387763 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386452    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-host-var-lib-cni-multus\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.387763 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386457    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-host-var-lib-cni-bin\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.387763 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386467    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-host-run-ovn-kubernetes\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.387763 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386477    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-host-slash\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.387763 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386482    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-host-run-netns\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.387763 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386491    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-host-var-lib-cni-multus\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.387763 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386501    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-host-run-netns\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.387763 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386503    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-host-cni-netd\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.387763 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386518    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-host-slash\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.387763 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386527    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-ovn-node-metrics-cert\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.387763 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386539    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-host-cni-netd\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.387763 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386546    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-blg28\" (UniqueName: \"kubernetes.io/projected/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-kube-api-access-blg28\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.388270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386582    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-etc-systemd\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.388270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386599    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-host-run-multus-certs\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.388270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386621    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-ovnkube-script-lib\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.388270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386653    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5f06d35-71cb-4bf8-af1b-e806af89003e-host-run-multus-certs\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.388270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386726    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ddb6ca29-0d32-41ce-a7d6-8e737ff47930-iptables-alerter-script\") pod \"iptables-alerter-fd2xn\" (UID: \"ddb6ca29-0d32-41ce-a7d6-8e737ff47930\") " pod="openshift-network-operator/iptables-alerter-fd2xn"
Apr 22 19:57:46.388270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386751    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-etc-sysctl-d\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.388270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386772    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-lib-modules\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.388270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386786    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-tmp\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.388270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386800    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Apr 22 19:57:46.388270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386800    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-var-lib-kubelet\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.388270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386866    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxvtq\" (UniqueName: \"kubernetes.io/projected/f12eca63-f2df-4781-97dd-7968beea64ba-kube-api-access-xxvtq\") pod \"node-resolver-rmblf\" (UID: \"f12eca63-f2df-4781-97dd-7968beea64ba\") " pod="openshift-dns/node-resolver-rmblf"
Apr 22 19:57:46.388270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.386887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d5d3cbed-7c79-4ea8-a02c-2992c80af473-agent-certs\") pod \"konnectivity-agent-pmrkj\" (UID: \"d5d3cbed-7c79-4ea8-a02c-2992c80af473\") " pod="kube-system/konnectivity-agent-pmrkj"
Apr 22 19:57:46.388270 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.387491    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-ovnkube-script-lib\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.390333 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.390320    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-ovn-node-metrics-cert\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.394055 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.394030    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsj7w\" (UniqueName: \"kubernetes.io/projected/ddb6ca29-0d32-41ce-a7d6-8e737ff47930-kube-api-access-jsj7w\") pod \"iptables-alerter-fd2xn\" (UID: \"ddb6ca29-0d32-41ce-a7d6-8e737ff47930\") " pod="openshift-network-operator/iptables-alerter-fd2xn"
Apr 22 19:57:46.394207 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.394191    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8r68\" (UniqueName: \"kubernetes.io/projected/671f3faa-9804-447f-acb1-14a0c42815dd-kube-api-access-c8r68\") pod \"node-ca-tn59d\" (UID: \"671f3faa-9804-447f-acb1-14a0c42815dd\") " pod="openshift-image-registry/node-ca-tn59d"
Apr 22 19:57:46.394481 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.394468    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmmj6\" (UniqueName: \"kubernetes.io/projected/d5f06d35-71cb-4bf8-af1b-e806af89003e-kube-api-access-tmmj6\") pod \"multus-rzjvv\" (UID: \"d5f06d35-71cb-4bf8-af1b-e806af89003e\") " pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.394518 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.394477    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdkcs\" (UniqueName: \"kubernetes.io/projected/dc0bc97c-7cea-48d0-a254-fb23e782e797-kube-api-access-cdkcs\") pod \"multus-additional-cni-plugins-q25vj\" (UID: \"dc0bc97c-7cea-48d0-a254-fb23e782e797\") " pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.394933 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.394918    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blg28\" (UniqueName: \"kubernetes.io/projected/ffc78e12-8bf6-4247-b1b7-b3da2c727db5-kube-api-access-blg28\") pod \"ovnkube-node-rbkp9\" (UID: \"ffc78e12-8bf6-4247-b1b7-b3da2c727db5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.419698 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.419676    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rhzw7"
Apr 22 19:57:46.427283 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.427266    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rhzw7"
Apr 22 19:57:46.487372 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.487331    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-etc-systemd\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.487703 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.487374    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-etc-sysctl-d\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.487773 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.487733    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-lib-modules\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.487824 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.487768    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-etc-systemd\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.487824 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.487780    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-tmp\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.487824 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.487793    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-etc-sysctl-d\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.489496 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.489465    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-var-lib-kubelet\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.489619 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.489527    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xxvtq\" (UniqueName: \"kubernetes.io/projected/f12eca63-f2df-4781-97dd-7968beea64ba-kube-api-access-xxvtq\") pod \"node-resolver-rmblf\" (UID: \"f12eca63-f2df-4781-97dd-7968beea64ba\") " pod="openshift-dns/node-resolver-rmblf"
Apr 22 19:57:46.489619 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.489608    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d5d3cbed-7c79-4ea8-a02c-2992c80af473-agent-certs\") pod \"konnectivity-agent-pmrkj\" (UID: \"d5d3cbed-7c79-4ea8-a02c-2992c80af473\") " pod="kube-system/konnectivity-agent-pmrkj"
Apr 22 19:57:46.489718 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.489615    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-lib-modules\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.489718 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.489629    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-var-lib-kubelet\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.489718 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.489638    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-etc-sysconfig\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.489718 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.489693    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-etc-sysconfig\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.489882 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.489731    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-etc-tuned\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.489882 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.489763    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs\") pod \"network-metrics-daemon-lkqdc\" (UID: \"7dd397be-1edf-42eb-8b45-a3291c8c3b7f\") " pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:57:46.489882 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.489845    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d5d3cbed-7c79-4ea8-a02c-2992c80af473-konnectivity-ca\") pod \"konnectivity-agent-pmrkj\" (UID: \"d5d3cbed-7c79-4ea8-a02c-2992c80af473\") " pod="kube-system/konnectivity-agent-pmrkj"
Apr 22 19:57:46.490011 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.489889    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34b46484-7515-431f-9460-068a93fec7e4-device-dir\") pod \"aws-ebs-csi-driver-node-65cmh\" (UID: \"34b46484-7515-431f-9460-068a93fec7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.490201 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:46.490183    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:46.490515 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:46.490502    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs podName:7dd397be-1edf-42eb-8b45-a3291c8c3b7f nodeName:}" failed. No retries permitted until 2026-04-22 19:57:46.99047464 +0000 UTC m=+2.090430899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs") pod "network-metrics-daemon-lkqdc" (UID: "7dd397be-1edf-42eb-8b45-a3291c8c3b7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:46.490678 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.490654    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d5d3cbed-7c79-4ea8-a02c-2992c80af473-konnectivity-ca\") pod \"konnectivity-agent-pmrkj\" (UID: \"d5d3cbed-7c79-4ea8-a02c-2992c80af473\") " pod="kube-system/konnectivity-agent-pmrkj"
Apr 22 19:57:46.490736 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.490722    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-etc-modprobe-d\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.490787 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.490761    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rj7k8\" (UniqueName: \"kubernetes.io/projected/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-kube-api-access-rj7k8\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.490867 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.490845    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34b46484-7515-431f-9460-068a93fec7e4-device-dir\") pod \"aws-ebs-csi-driver-node-65cmh\" (UID: \"34b46484-7515-431f-9460-068a93fec7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.490867 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.490847    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9ch42\" (UniqueName: \"kubernetes.io/projected/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-kube-api-access-9ch42\") pod \"network-metrics-daemon-lkqdc\" (UID: \"7dd397be-1edf-42eb-8b45-a3291c8c3b7f\") " pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:57:46.490969 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.490906    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34b46484-7515-431f-9460-068a93fec7e4-sys-fs\") pod \"aws-ebs-csi-driver-node-65cmh\" (UID: \"34b46484-7515-431f-9460-068a93fec7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.490969 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.490947    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-etc-sysctl-conf\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.491054 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.490982    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-host\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.491054 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.490989    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-etc-modprobe-d\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.491054 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491015    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f12eca63-f2df-4781-97dd-7968beea64ba-hosts-file\") pod \"node-resolver-rmblf\" (UID: \"f12eca63-f2df-4781-97dd-7968beea64ba\") " pod="openshift-dns/node-resolver-rmblf"
Apr 22 19:57:46.491223 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491070    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34b46484-7515-431f-9460-068a93fec7e4-socket-dir\") pod \"aws-ebs-csi-driver-node-65cmh\" (UID: \"34b46484-7515-431f-9460-068a93fec7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.491223 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491083    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34b46484-7515-431f-9460-068a93fec7e4-sys-fs\") pod \"aws-ebs-csi-driver-node-65cmh\" (UID: \"34b46484-7515-431f-9460-068a93fec7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.491223 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491137    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-etc-sysctl-conf\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.491363 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491219    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-host\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.491363 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491250    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f12eca63-f2df-4781-97dd-7968beea64ba-hosts-file\") pod \"node-resolver-rmblf\" (UID: \"f12eca63-f2df-4781-97dd-7968beea64ba\") " pod="openshift-dns/node-resolver-rmblf"
Apr 22 19:57:46.491363 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491251    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f12eca63-f2df-4781-97dd-7968beea64ba-tmp-dir\") pod \"node-resolver-rmblf\" (UID: \"f12eca63-f2df-4781-97dd-7968beea64ba\") " pod="openshift-dns/node-resolver-rmblf"
Apr 22 19:57:46.491363 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491299    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34b46484-7515-431f-9460-068a93fec7e4-registration-dir\") pod \"aws-ebs-csi-driver-node-65cmh\" (UID: \"34b46484-7515-431f-9460-068a93fec7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.491363 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491344    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-run\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.491363 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491356    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34b46484-7515-431f-9460-068a93fec7e4-socket-dir\") pod \"aws-ebs-csi-driver-node-65cmh\" (UID: \"34b46484-7515-431f-9460-068a93fec7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.491624 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491376    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j8vfm\" (UniqueName: \"kubernetes.io/projected/34b46484-7515-431f-9460-068a93fec7e4-kube-api-access-j8vfm\") pod \"aws-ebs-csi-driver-node-65cmh\" (UID: \"34b46484-7515-431f-9460-068a93fec7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.491624 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491422    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8tfnx\" (UniqueName: \"kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx\") pod \"network-check-target-pq8s7\" (UID: \"7f4a64a7-a0cc-426a-9469-5cb2654b3b1d\") " pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:57:46.491624 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491427    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-run\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.491624 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491421    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34b46484-7515-431f-9460-068a93fec7e4-registration-dir\") pod \"aws-ebs-csi-driver-node-65cmh\" (UID: \"34b46484-7515-431f-9460-068a93fec7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.491624 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491460    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-sys\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.491624 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491527    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-sys\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.491624 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491519    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-etc-kubernetes\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.491624 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491590    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-etc-kubernetes\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.491624 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491606    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f12eca63-f2df-4781-97dd-7968beea64ba-tmp-dir\") pod \"node-resolver-rmblf\" (UID: \"f12eca63-f2df-4781-97dd-7968beea64ba\") " pod="openshift-dns/node-resolver-rmblf"
Apr 22 19:57:46.492003 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491650    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34b46484-7515-431f-9460-068a93fec7e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-65cmh\" (UID: \"34b46484-7515-431f-9460-068a93fec7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.492003 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491685    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34b46484-7515-431f-9460-068a93fec7e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-65cmh\" (UID: \"34b46484-7515-431f-9460-068a93fec7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.492003 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491857    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34b46484-7515-431f-9460-068a93fec7e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-65cmh\" (UID: \"34b46484-7515-431f-9460-068a93fec7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.492003 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.491857    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34b46484-7515-431f-9460-068a93fec7e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-65cmh\" (UID: \"34b46484-7515-431f-9460-068a93fec7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.492641 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.492618    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-tmp\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.492715 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.492652    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-etc-tuned\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.492767 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.492748    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d5d3cbed-7c79-4ea8-a02c-2992c80af473-agent-certs\") pod \"konnectivity-agent-pmrkj\" (UID: \"d5d3cbed-7c79-4ea8-a02c-2992c80af473\") " pod="kube-system/konnectivity-agent-pmrkj"
Apr 22 19:57:46.499060 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:46.499040    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 22 19:57:46.499060 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:46.499063    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 22 19:57:46.499234 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:46.499075    2576 projected.go:194] Error preparing data for projected volume kube-api-access-8tfnx for pod openshift-network-diagnostics/network-check-target-pq8s7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:46.499234 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:46.499142    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx podName:7f4a64a7-a0cc-426a-9469-5cb2654b3b1d nodeName:}" failed. No retries permitted until 2026-04-22 19:57:46.999124468 +0000 UTC m=+2.099080728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8tfnx" (UniqueName: "kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx") pod "network-check-target-pq8s7" (UID: "7f4a64a7-a0cc-426a-9469-5cb2654b3b1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:46.501230 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.501212    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8vfm\" (UniqueName: \"kubernetes.io/projected/34b46484-7515-431f-9460-068a93fec7e4-kube-api-access-j8vfm\") pod \"aws-ebs-csi-driver-node-65cmh\" (UID: \"34b46484-7515-431f-9460-068a93fec7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.501388 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.501368    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ch42\" (UniqueName: \"kubernetes.io/projected/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-kube-api-access-9ch42\") pod \"network-metrics-daemon-lkqdc\" (UID: \"7dd397be-1edf-42eb-8b45-a3291c8c3b7f\") " pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:57:46.501680 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.501658    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxvtq\" (UniqueName: \"kubernetes.io/projected/f12eca63-f2df-4781-97dd-7968beea64ba-kube-api-access-xxvtq\") pod \"node-resolver-rmblf\" (UID: \"f12eca63-f2df-4781-97dd-7968beea64ba\") " pod="openshift-dns/node-resolver-rmblf"
Apr 22 19:57:46.501843 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.501827    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj7k8\" (UniqueName: \"kubernetes.io/projected/6cd0be72-50fc-4900-a3d9-c4a7b8ad9256-kube-api-access-rj7k8\") pod \"tuned-vzjns\" (UID: \"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256\") " pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.596769 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.596697    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rzjvv"
Apr 22 19:57:46.602306 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:46.602286    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f06d35_71cb_4bf8_af1b_e806af89003e.slice/crio-02229c67755cc077f4dd37de8892a60405e336607ac7a0e6a574f27e10bbb283 WatchSource:0}: Error finding container 02229c67755cc077f4dd37de8892a60405e336607ac7a0e6a574f27e10bbb283: Status 404 returned error can't find the container with id 02229c67755cc077f4dd37de8892a60405e336607ac7a0e6a574f27e10bbb283
Apr 22 19:57:46.603675 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.603658    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fd2xn"
Apr 22 19:57:46.609848 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:46.609827    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb6ca29_0d32_41ce_a7d6_8e737ff47930.slice/crio-e143987bdd656d388c592b65a379672094110770a231b59a4a221c889445f1af WatchSource:0}: Error finding container e143987bdd656d388c592b65a379672094110770a231b59a4a221c889445f1af: Status 404 returned error can't find the container with id e143987bdd656d388c592b65a379672094110770a231b59a4a221c889445f1af
Apr 22 19:57:46.610575 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.610558    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:57:46.614777 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.614760    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tn59d"
Apr 22 19:57:46.616963 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:46.616938    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc78e12_8bf6_4247_b1b7_b3da2c727db5.slice/crio-6046ee849234db664d36972d1edac946b7bb08f055f139a319f3038379f734da WatchSource:0}: Error finding container 6046ee849234db664d36972d1edac946b7bb08f055f139a319f3038379f734da: Status 404 returned error can't find the container with id 6046ee849234db664d36972d1edac946b7bb08f055f139a319f3038379f734da
Apr 22 19:57:46.620967 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.620952    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q25vj"
Apr 22 19:57:46.621859 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:46.621837    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod671f3faa_9804_447f_acb1_14a0c42815dd.slice/crio-afd0adff39adbcd967a41242af7a43be0a5c689d9d35e0240935e5c25ea8d560 WatchSource:0}: Error finding container afd0adff39adbcd967a41242af7a43be0a5c689d9d35e0240935e5c25ea8d560: Status 404 returned error can't find the container with id afd0adff39adbcd967a41242af7a43be0a5c689d9d35e0240935e5c25ea8d560
Apr 22 19:57:46.626233 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.626216    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pmrkj"
Apr 22 19:57:46.626690 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:46.626668    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0bc97c_7cea_48d0_a254_fb23e782e797.slice/crio-d8c96fe1d67989d117333ea969feec2ea9c0668d4a7cc7313497f87d1172e9d9 WatchSource:0}: Error finding container d8c96fe1d67989d117333ea969feec2ea9c0668d4a7cc7313497f87d1172e9d9: Status 404 returned error can't find the container with id d8c96fe1d67989d117333ea969feec2ea9c0668d4a7cc7313497f87d1172e9d9
Apr 22 19:57:46.631794 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.631775    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh"
Apr 22 19:57:46.632332 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:46.632182    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d3cbed_7c79_4ea8_a02c_2992c80af473.slice/crio-160cac8c9bd1e03709c3d425db2c8aa18c2e21bd177fad86e98858143243ffc5 WatchSource:0}: Error finding container 160cac8c9bd1e03709c3d425db2c8aa18c2e21bd177fad86e98858143243ffc5: Status 404 returned error can't find the container with id 160cac8c9bd1e03709c3d425db2c8aa18c2e21bd177fad86e98858143243ffc5
Apr 22 19:57:46.640269 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:46.638719    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b46484_7515_431f_9460_068a93fec7e4.slice/crio-d333272dfac29b91341765fef2450e70861b3308e9f6d0cf73cd257fa84e4ee9 WatchSource:0}: Error finding container d333272dfac29b91341765fef2450e70861b3308e9f6d0cf73cd257fa84e4ee9: Status 404 returned error can't find the container with id d333272dfac29b91341765fef2450e70861b3308e9f6d0cf73cd257fa84e4ee9
Apr 22 19:57:46.640269 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.638807    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vzjns"
Apr 22 19:57:46.643295 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.643167    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rmblf"
Apr 22 19:57:46.647069 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:46.647051    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd0be72_50fc_4900_a3d9_c4a7b8ad9256.slice/crio-26dc0d07f468a08a70ab7f51c931b5c31413b4d43465d3994689ba994614d0d1 WatchSource:0}: Error finding container 26dc0d07f468a08a70ab7f51c931b5c31413b4d43465d3994689ba994614d0d1: Status 404 returned error can't find the container with id 26dc0d07f468a08a70ab7f51c931b5c31413b4d43465d3994689ba994614d0d1
Apr 22 19:57:46.650413 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:57:46.650390    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12eca63_f2df_4781_97dd_7968beea64ba.slice/crio-faf41fed45531a00fe8b1ba2886ddbf91cb48e291dd174ad9288731f6dea7a2f WatchSource:0}: Error finding container faf41fed45531a00fe8b1ba2886ddbf91cb48e291dd174ad9288731f6dea7a2f: Status 404 returned error can't find the container with id faf41fed45531a00fe8b1ba2886ddbf91cb48e291dd174ad9288731f6dea7a2f
Apr 22 19:57:46.995196 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:46.995161    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs\") pod \"network-metrics-daemon-lkqdc\" (UID: \"7dd397be-1edf-42eb-8b45-a3291c8c3b7f\") " pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:57:46.995412 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:46.995303    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:46.995412 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:46.995365    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs podName:7dd397be-1edf-42eb-8b45-a3291c8c3b7f nodeName:}" failed. No retries permitted until 2026-04-22 19:57:47.995344869 +0000 UTC m=+3.095301120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs") pod "network-metrics-daemon-lkqdc" (UID: "7dd397be-1edf-42eb-8b45-a3291c8c3b7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:47.096269 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:47.096203    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8tfnx\" (UniqueName: \"kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx\") pod \"network-check-target-pq8s7\" (UID: \"7f4a64a7-a0cc-426a-9469-5cb2654b3b1d\") " pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:57:47.096435 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:47.096385    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 22 19:57:47.096435 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:47.096404    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 22 19:57:47.096435 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:47.096416    2576 projected.go:194] Error preparing data for projected volume kube-api-access-8tfnx for pod openshift-network-diagnostics/network-check-target-pq8s7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:47.096638 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:47.096486    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx podName:7f4a64a7-a0cc-426a-9469-5cb2654b3b1d nodeName:}" failed. No retries permitted until 2026-04-22 19:57:48.096455286 +0000 UTC m=+3.196411534 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8tfnx" (UniqueName: "kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx") pod "network-check-target-pq8s7" (UID: "7f4a64a7-a0cc-426a-9469-5cb2654b3b1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:47.394833 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:47.394783    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh" event={"ID":"34b46484-7515-431f-9460-068a93fec7e4","Type":"ContainerStarted","Data":"d333272dfac29b91341765fef2450e70861b3308e9f6d0cf73cd257fa84e4ee9"}
Apr 22 19:57:47.414946 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:47.414871    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tn59d" event={"ID":"671f3faa-9804-447f-acb1-14a0c42815dd","Type":"ContainerStarted","Data":"afd0adff39adbcd967a41242af7a43be0a5c689d9d35e0240935e5c25ea8d560"}
Apr 22 19:57:47.428897 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:47.428825    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-21 19:52:46 +0000 UTC" deadline="2027-10-31 03:35:49.781456745 +0000 UTC"
Apr 22 19:57:47.428897 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:47.428855    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13351h38m2.352605787s"
Apr 22 19:57:47.429779 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:47.429735    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fd2xn" event={"ID":"ddb6ca29-0d32-41ce-a7d6-8e737ff47930","Type":"ContainerStarted","Data":"e143987bdd656d388c592b65a379672094110770a231b59a4a221c889445f1af"}
Apr 22 19:57:47.446966 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:47.446933    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rmblf" event={"ID":"f12eca63-f2df-4781-97dd-7968beea64ba","Type":"ContainerStarted","Data":"faf41fed45531a00fe8b1ba2886ddbf91cb48e291dd174ad9288731f6dea7a2f"}
Apr 22 19:57:47.459474 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:47.459426    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pmrkj" event={"ID":"d5d3cbed-7c79-4ea8-a02c-2992c80af473","Type":"ContainerStarted","Data":"160cac8c9bd1e03709c3d425db2c8aa18c2e21bd177fad86e98858143243ffc5"}
Apr 22 19:57:47.485220 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:47.485181    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q25vj" event={"ID":"dc0bc97c-7cea-48d0-a254-fb23e782e797","Type":"ContainerStarted","Data":"d8c96fe1d67989d117333ea969feec2ea9c0668d4a7cc7313497f87d1172e9d9"}
Apr 22 19:57:47.494970 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:47.494941    2576 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 22 19:57:47.498046 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:47.498007    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9" event={"ID":"ffc78e12-8bf6-4247-b1b7-b3da2c727db5","Type":"ContainerStarted","Data":"6046ee849234db664d36972d1edac946b7bb08f055f139a319f3038379f734da"}
Apr 22 19:57:47.509447 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:47.509415    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rzjvv" event={"ID":"d5f06d35-71cb-4bf8-af1b-e806af89003e","Type":"ContainerStarted","Data":"02229c67755cc077f4dd37de8892a60405e336607ac7a0e6a574f27e10bbb283"}
Apr 22 19:57:47.524251 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:47.524216    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vzjns" event={"ID":"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256","Type":"ContainerStarted","Data":"26dc0d07f468a08a70ab7f51c931b5c31413b4d43465d3994689ba994614d0d1"}
Apr 22 19:57:48.003098 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:48.003021    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs\") pod \"network-metrics-daemon-lkqdc\" (UID: \"7dd397be-1edf-42eb-8b45-a3291c8c3b7f\") " pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:57:48.003300 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:48.003206    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:48.003300 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:48.003270    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs podName:7dd397be-1edf-42eb-8b45-a3291c8c3b7f nodeName:}" failed. No retries permitted until 2026-04-22 19:57:50.003251482 +0000 UTC m=+5.103207731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs") pod "network-metrics-daemon-lkqdc" (UID: "7dd397be-1edf-42eb-8b45-a3291c8c3b7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:48.103623 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:48.103586    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8tfnx\" (UniqueName: \"kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx\") pod \"network-check-target-pq8s7\" (UID: \"7f4a64a7-a0cc-426a-9469-5cb2654b3b1d\") " pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:57:48.103828 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:48.103792    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 22 19:57:48.103828 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:48.103814    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 22 19:57:48.103828 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:48.103827    2576 projected.go:194] Error preparing data for projected volume kube-api-access-8tfnx for pod openshift-network-diagnostics/network-check-target-pq8s7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:48.104019 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:48.103886    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx podName:7f4a64a7-a0cc-426a-9469-5cb2654b3b1d nodeName:}" failed. No retries permitted until 2026-04-22 19:57:50.103866632 +0000 UTC m=+5.203822879 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8tfnx" (UniqueName: "kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx") pod "network-check-target-pq8s7" (UID: "7f4a64a7-a0cc-426a-9469-5cb2654b3b1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:48.172399 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:48.172363    2576 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 22 19:57:48.369477 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:48.369444    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:57:48.369661 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:48.369602    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkqdc" podUID="7dd397be-1edf-42eb-8b45-a3291c8c3b7f"
Apr 22 19:57:48.370036 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:48.370015    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:57:48.370130 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:48.370110    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pq8s7" podUID="7f4a64a7-a0cc-426a-9469-5cb2654b3b1d"
Apr 22 19:57:48.429906 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:48.429842    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-21 19:52:46 +0000 UTC" deadline="2027-10-16 19:43:19.080706188 +0000 UTC"
Apr 22 19:57:48.429906 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:48.429878    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13007h45m30.650831438s"
Apr 22 19:57:50.020494 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:50.020454    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs\") pod \"network-metrics-daemon-lkqdc\" (UID: \"7dd397be-1edf-42eb-8b45-a3291c8c3b7f\") " pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:57:50.020947 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:50.020664    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:50.020947 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:50.020726    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs podName:7dd397be-1edf-42eb-8b45-a3291c8c3b7f nodeName:}" failed. No retries permitted until 2026-04-22 19:57:54.020708444 +0000 UTC m=+9.120664693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs") pod "network-metrics-daemon-lkqdc" (UID: "7dd397be-1edf-42eb-8b45-a3291c8c3b7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:50.121396 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:50.121306    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8tfnx\" (UniqueName: \"kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx\") pod \"network-check-target-pq8s7\" (UID: \"7f4a64a7-a0cc-426a-9469-5cb2654b3b1d\") " pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:57:50.121652 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:50.121551    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 22 19:57:50.121652 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:50.121571    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 22 19:57:50.121652 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:50.121583    2576 projected.go:194] Error preparing data for projected volume kube-api-access-8tfnx for pod openshift-network-diagnostics/network-check-target-pq8s7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:50.121652 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:50.121648    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx podName:7f4a64a7-a0cc-426a-9469-5cb2654b3b1d nodeName:}" failed. No retries permitted until 2026-04-22 19:57:54.121629546 +0000 UTC m=+9.221585803 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8tfnx" (UniqueName: "kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx") pod "network-check-target-pq8s7" (UID: "7f4a64a7-a0cc-426a-9469-5cb2654b3b1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:50.369471 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:50.369442    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:57:50.369664 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:50.369485    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:57:50.369664 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:50.369590    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkqdc" podUID="7dd397be-1edf-42eb-8b45-a3291c8c3b7f"
Apr 22 19:57:50.369952 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:50.369930    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pq8s7" podUID="7f4a64a7-a0cc-426a-9469-5cb2654b3b1d"
Apr 22 19:57:52.369951 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:52.369337    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:57:52.369951 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:52.369481    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pq8s7" podUID="7f4a64a7-a0cc-426a-9469-5cb2654b3b1d"
Apr 22 19:57:52.369951 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:52.369344    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:57:52.369951 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:52.369614    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkqdc" podUID="7dd397be-1edf-42eb-8b45-a3291c8c3b7f"
Apr 22 19:57:54.053294 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:54.053255    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs\") pod \"network-metrics-daemon-lkqdc\" (UID: \"7dd397be-1edf-42eb-8b45-a3291c8c3b7f\") " pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:57:54.053743 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:54.053406    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:54.053743 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:54.053466    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs podName:7dd397be-1edf-42eb-8b45-a3291c8c3b7f nodeName:}" failed. No retries permitted until 2026-04-22 19:58:02.053452022 +0000 UTC m=+17.153408268 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs") pod "network-metrics-daemon-lkqdc" (UID: "7dd397be-1edf-42eb-8b45-a3291c8c3b7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:54.154996 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:54.154269    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8tfnx\" (UniqueName: \"kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx\") pod \"network-check-target-pq8s7\" (UID: \"7f4a64a7-a0cc-426a-9469-5cb2654b3b1d\") " pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:57:54.154996 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:54.154466    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 22 19:57:54.154996 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:54.154483    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 22 19:57:54.154996 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:54.154494    2576 projected.go:194] Error preparing data for projected volume kube-api-access-8tfnx for pod openshift-network-diagnostics/network-check-target-pq8s7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:54.154996 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:54.154549    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx podName:7f4a64a7-a0cc-426a-9469-5cb2654b3b1d nodeName:}" failed. No retries permitted until 2026-04-22 19:58:02.154531602 +0000 UTC m=+17.254487845 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8tfnx" (UniqueName: "kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx") pod "network-check-target-pq8s7" (UID: "7f4a64a7-a0cc-426a-9469-5cb2654b3b1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:54.370878 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:54.370036    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:57:54.370878 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:54.370411    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkqdc" podUID="7dd397be-1edf-42eb-8b45-a3291c8c3b7f"
Apr 22 19:57:54.370878 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:54.370231    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:57:54.370878 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:54.370805    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pq8s7" podUID="7f4a64a7-a0cc-426a-9469-5cb2654b3b1d"
Apr 22 19:57:56.369866 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:56.369827    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:57:56.370337 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:56.369968    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkqdc" podUID="7dd397be-1edf-42eb-8b45-a3291c8c3b7f"
Apr 22 19:57:56.370337 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:56.369999    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:57:56.370337 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:56.370111    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pq8s7" podUID="7f4a64a7-a0cc-426a-9469-5cb2654b3b1d"
Apr 22 19:57:58.369312 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:58.369260    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:57:58.369312 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:57:58.369299    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:57:58.369812 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:58.369390    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkqdc" podUID="7dd397be-1edf-42eb-8b45-a3291c8c3b7f"
Apr 22 19:57:58.369812 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:57:58.369528    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pq8s7" podUID="7f4a64a7-a0cc-426a-9469-5cb2654b3b1d"
Apr 22 19:58:00.369672 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:00.369635    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:58:00.370111 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:00.369635    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:58:00.370111 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:00.369752    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pq8s7" podUID="7f4a64a7-a0cc-426a-9469-5cb2654b3b1d"
Apr 22 19:58:00.370111 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:00.369839    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkqdc" podUID="7dd397be-1edf-42eb-8b45-a3291c8c3b7f"
Apr 22 19:58:02.114986 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:02.114936    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs\") pod \"network-metrics-daemon-lkqdc\" (UID: \"7dd397be-1edf-42eb-8b45-a3291c8c3b7f\") " pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:58:02.115457 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:02.115085    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:58:02.115457 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:02.115178    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs podName:7dd397be-1edf-42eb-8b45-a3291c8c3b7f nodeName:}" failed. No retries permitted until 2026-04-22 19:58:18.115159438 +0000 UTC m=+33.215115693 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs") pod "network-metrics-daemon-lkqdc" (UID: "7dd397be-1edf-42eb-8b45-a3291c8c3b7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:58:02.216167 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:02.216115    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8tfnx\" (UniqueName: \"kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx\") pod \"network-check-target-pq8s7\" (UID: \"7f4a64a7-a0cc-426a-9469-5cb2654b3b1d\") " pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:58:02.216339 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:02.216315    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 22 19:58:02.216394 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:02.216341    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 22 19:58:02.216394 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:02.216354    2576 projected.go:194] Error preparing data for projected volume kube-api-access-8tfnx for pod openshift-network-diagnostics/network-check-target-pq8s7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:58:02.216503 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:02.216417    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx podName:7f4a64a7-a0cc-426a-9469-5cb2654b3b1d nodeName:}" failed. No retries permitted until 2026-04-22 19:58:18.216397862 +0000 UTC m=+33.316354125 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8tfnx" (UniqueName: "kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx") pod "network-check-target-pq8s7" (UID: "7f4a64a7-a0cc-426a-9469-5cb2654b3b1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:58:02.369436 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:02.369411    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:58:02.369616 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:02.369411    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:58:02.369616 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:02.369552    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkqdc" podUID="7dd397be-1edf-42eb-8b45-a3291c8c3b7f"
Apr 22 19:58:02.369741 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:02.369617    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pq8s7" podUID="7f4a64a7-a0cc-426a-9469-5cb2654b3b1d"
Apr 22 19:58:04.369560 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:04.369524    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:58:04.369560 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:04.369571    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:58:04.370141 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:04.369685    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkqdc" podUID="7dd397be-1edf-42eb-8b45-a3291c8c3b7f"
Apr 22 19:58:04.370141 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:04.369838    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pq8s7" podUID="7f4a64a7-a0cc-426a-9469-5cb2654b3b1d"
Apr 22 19:58:05.561420 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:05.561255    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9" event={"ID":"ffc78e12-8bf6-4247-b1b7-b3da2c727db5","Type":"ContainerStarted","Data":"9159e8d882b1a0aed1fb5952b43e0f78b7ed8e28868435c9e7987044e3ae9775"}
Apr 22 19:58:05.562105 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:05.561429    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9" event={"ID":"ffc78e12-8bf6-4247-b1b7-b3da2c727db5","Type":"ContainerStarted","Data":"eab6c7b180357024ee8aad3c0bdb1cef70ce5b774c8e922d3e60d0a675735c05"}
Apr 22 19:58:05.562105 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:05.561439    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9" event={"ID":"ffc78e12-8bf6-4247-b1b7-b3da2c727db5","Type":"ContainerStarted","Data":"e7abee5bf2501d0d619325f022e59e40e9a5ef16b588b4501cacd3ccbcce00a1"}
Apr 22 19:58:05.562105 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:05.561448    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9" event={"ID":"ffc78e12-8bf6-4247-b1b7-b3da2c727db5","Type":"ContainerStarted","Data":"7009b74d1941060a3c94879cf283c2ec8d7c1e9564f9e411e4a76d31e8e6d3ce"}
Apr 22 19:58:05.562105 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:05.561458    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9" event={"ID":"ffc78e12-8bf6-4247-b1b7-b3da2c727db5","Type":"ContainerStarted","Data":"36e8ade6383daf1d1f11b823d4c4bad89a79830849fadffd2d4f243581651cf1"}
Apr 22 19:58:05.563221 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:05.563196    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rzjvv" event={"ID":"d5f06d35-71cb-4bf8-af1b-e806af89003e","Type":"ContainerStarted","Data":"1662f41fc9163acc69b1e6227963a391ca121d39bb954b141bb8cb35a67c4ded"}
Apr 22 19:58:05.564421 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:05.564398    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vzjns" event={"ID":"6cd0be72-50fc-4900-a3d9-c4a7b8ad9256","Type":"ContainerStarted","Data":"278d473695f3036f901e49bbe3e2d5aaad370358a038f2b54d8f5e69761b9b5d"}
Apr 22 19:58:05.566432 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:05.566091    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-245.ec2.internal" event={"ID":"0353ff154730f2cf2a2ea68e586089e5","Type":"ContainerStarted","Data":"ec7cc8079ddecc3e26ea2e1348f1119b96c3295aebd27a975d9bfd11c3d8977c"}
Apr 22 19:58:05.582465 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:05.582404    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-rzjvv" podStartSLOduration=2.549975341 podStartE2EDuration="20.582389281s" podCreationTimestamp="2026-04-22 19:57:45 +0000 UTC" firstStartedPulling="2026-04-22 19:57:46.603783449 +0000 UTC m=+1.703739722" lastFinishedPulling="2026-04-22 19:58:04.636197403 +0000 UTC m=+19.736153662" observedRunningTime="2026-04-22 19:58:05.581215271 +0000 UTC m=+20.681171535" watchObservedRunningTime="2026-04-22 19:58:05.582389281 +0000 UTC m=+20.682345547"
Apr 22 19:58:05.596243 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:05.596173    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-245.ec2.internal" podStartSLOduration=19.596135636 podStartE2EDuration="19.596135636s" podCreationTimestamp="2026-04-22 19:57:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 19:58:05.596112559 +0000 UTC m=+20.696068842" watchObservedRunningTime="2026-04-22 19:58:05.596135636 +0000 UTC m=+20.696091904"
Apr 22 19:58:06.370295 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:06.370106    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:58:06.370440 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:06.370118    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:58:06.370440 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:06.370379    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pq8s7" podUID="7f4a64a7-a0cc-426a-9469-5cb2654b3b1d"
Apr 22 19:58:06.370521 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:06.370489    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkqdc" podUID="7dd397be-1edf-42eb-8b45-a3291c8c3b7f"
Apr 22 19:58:06.568920 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:06.568891    2576 generic.go:358] "Generic (PLEG): container finished" podID="dc0bc97c-7cea-48d0-a254-fb23e782e797" containerID="c0041e991d83b965aa58b3446489ab08c34a7b3d6baed1fa3c599a086147df04" exitCode=0
Apr 22 19:58:06.569294 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:06.568980    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q25vj" event={"ID":"dc0bc97c-7cea-48d0-a254-fb23e782e797","Type":"ContainerDied","Data":"c0041e991d83b965aa58b3446489ab08c34a7b3d6baed1fa3c599a086147df04"}
Apr 22 19:58:06.572775 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:06.572752    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9" event={"ID":"ffc78e12-8bf6-4247-b1b7-b3da2c727db5","Type":"ContainerStarted","Data":"58a3f4488dc9e85d099299c1b55d579560b842fd2f8bfce78f1c0cb814d675f1"}
Apr 22 19:58:06.574289 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:06.574267    2576 generic.go:358] "Generic (PLEG): container finished" podID="7eff41dfcc7a8a8786c50b0bcb949e70" containerID="374e1df5cb809608a2facfaa54c65ae6d5f80ffafbc74f8e17ec17ea78a68e39" exitCode=0
Apr 22 19:58:06.574366 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:06.574318    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-245.ec2.internal" event={"ID":"7eff41dfcc7a8a8786c50b0bcb949e70","Type":"ContainerDied","Data":"374e1df5cb809608a2facfaa54c65ae6d5f80ffafbc74f8e17ec17ea78a68e39"}
Apr 22 19:58:06.575975 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:06.575951    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh" event={"ID":"34b46484-7515-431f-9460-068a93fec7e4","Type":"ContainerStarted","Data":"5869c2f94fd72c74a2613342572776f06393fb9b4048ea35c15f7e94db373cf0"}
Apr 22 19:58:06.578800 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:06.578776    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tn59d" event={"ID":"671f3faa-9804-447f-acb1-14a0c42815dd","Type":"ContainerStarted","Data":"7639146bfee9a63a88a1456f4e00106ac823b4f1c4f0abe78743d3d13f5f8874"}
Apr 22 19:58:06.581330 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:06.581308    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fd2xn" event={"ID":"ddb6ca29-0d32-41ce-a7d6-8e737ff47930","Type":"ContainerStarted","Data":"5767fa3528cfbdf04c7618bd75b18e7b83da1baf6d12292b67600110aef22802"}
Apr 22 19:58:06.582533 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:06.582504    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rmblf" event={"ID":"f12eca63-f2df-4781-97dd-7968beea64ba","Type":"ContainerStarted","Data":"e697b5c9543bc186c8802d3dbf4faad9f055da4d4ea939868b2ce27274bd1301"}
Apr 22 19:58:06.583744 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:06.583718    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pmrkj" event={"ID":"d5d3cbed-7c79-4ea8-a02c-2992c80af473","Type":"ContainerStarted","Data":"676faf31a7e5af15e7916476ad97a9cd01b6a6910e4a2715dbe19ccf4a785ae0"}
Apr 22 19:58:06.605567 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:06.605513    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vzjns" podStartSLOduration=3.62896245 podStartE2EDuration="21.605479499s" podCreationTimestamp="2026-04-22 19:57:45 +0000 UTC" firstStartedPulling="2026-04-22 19:57:46.648855965 +0000 UTC m=+1.748812221" lastFinishedPulling="2026-04-22 19:58:04.625373011 +0000 UTC m=+19.725329270" observedRunningTime="2026-04-22 19:58:05.613437034 +0000 UTC m=+20.713393335" watchObservedRunningTime="2026-04-22 19:58:06.605479499 +0000 UTC m=+21.705435766"
Apr 22 19:58:06.635656 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:06.635601    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-rmblf" podStartSLOduration=3.664094743 podStartE2EDuration="21.635582946s" podCreationTimestamp="2026-04-22 19:57:45 +0000 UTC" firstStartedPulling="2026-04-22 19:57:46.651941115 +0000 UTC m=+1.751897372" lastFinishedPulling="2026-04-22 19:58:04.623429315 +0000 UTC m=+19.723385575" observedRunningTime="2026-04-22 19:58:06.620833593 +0000 UTC m=+21.720789859" watchObservedRunningTime="2026-04-22 19:58:06.635582946 +0000 UTC m=+21.735539210"
Apr 22 19:58:06.648980 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:06.648940    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fd2xn" podStartSLOduration=3.6308922150000003 podStartE2EDuration="21.64892897s" podCreationTimestamp="2026-04-22 19:57:45 +0000 UTC" firstStartedPulling="2026-04-22 19:57:46.612360293 +0000 UTC m=+1.712316536" lastFinishedPulling="2026-04-22 19:58:04.630397037 +0000 UTC m=+19.730353291" observedRunningTime="2026-04-22 19:58:06.648383076 +0000 UTC m=+21.748339355" watchObservedRunningTime="2026-04-22 19:58:06.64892897 +0000 UTC m=+21.748885234"
Apr 22 19:58:06.665558 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:06.663969    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tn59d" podStartSLOduration=8.163813347 podStartE2EDuration="21.663950809s" podCreationTimestamp="2026-04-22 19:57:45 +0000 UTC" firstStartedPulling="2026-04-22 19:57:46.623814785 +0000 UTC m=+1.723771044" lastFinishedPulling="2026-04-22 19:58:00.123952264 +0000 UTC m=+15.223908506" observedRunningTime="2026-04-22 19:58:06.662704924 +0000 UTC m=+21.762661201" watchObservedRunningTime="2026-04-22 19:58:06.663950809 +0000 UTC m=+21.763907075"
Apr 22 19:58:06.688132 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:06.687984    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pmrkj" podStartSLOduration=3.720483169 podStartE2EDuration="21.687968818s" podCreationTimestamp="2026-04-22 19:57:45 +0000 UTC" firstStartedPulling="2026-04-22 19:57:46.63440483 +0000 UTC m=+1.734361074" lastFinishedPulling="2026-04-22 19:58:04.60189048 +0000 UTC m=+19.701846723" observedRunningTime="2026-04-22 19:58:06.687849189 +0000 UTC m=+21.787805475" watchObservedRunningTime="2026-04-22 19:58:06.687968818 +0000 UTC m=+21.787925083"
Apr 22 19:58:06.698645 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:06.698621    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Apr 22 19:58:07.355003 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:07.354896    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-04-22T19:58:06.698642637Z","UUID":"18768e48-991d-4d21-a2cf-5f71d985a766","Handler":null,"Name":"","Endpoint":""}
Apr 22 19:58:07.357005 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:07.356983    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Apr 22 19:58:07.357005 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:07.357009    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Apr 22 19:58:07.588581 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:07.588552    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-245.ec2.internal" event={"ID":"7eff41dfcc7a8a8786c50b0bcb949e70","Type":"ContainerStarted","Data":"b93672ae011da88c934b810c28fc98a7e705d6c688b66906532d942ef5cf429d"}
Apr 22 19:58:07.590214 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:07.590186    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh" event={"ID":"34b46484-7515-431f-9460-068a93fec7e4","Type":"ContainerStarted","Data":"d39e5d628e9544856d01ecb50e6aa083279f4dee128d69735ad32c5c349b458e"}
Apr 22 19:58:07.603231 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:07.603183    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-245.ec2.internal" podStartSLOduration=21.603165276 podStartE2EDuration="21.603165276s" podCreationTimestamp="2026-04-22 19:57:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 19:58:07.602838188 +0000 UTC m=+22.702794453" watchObservedRunningTime="2026-04-22 19:58:07.603165276 +0000 UTC m=+22.703121533"
Apr 22 19:58:08.370128 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:08.370095    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:58:08.370318 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:08.370094    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:58:08.370318 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:08.370258    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pq8s7" podUID="7f4a64a7-a0cc-426a-9469-5cb2654b3b1d"
Apr 22 19:58:08.370419 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:08.370330    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkqdc" podUID="7dd397be-1edf-42eb-8b45-a3291c8c3b7f"
Apr 22 19:58:08.375259 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:08.375231    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pmrkj"
Apr 22 19:58:08.375818 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:08.375795    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pmrkj"
Apr 22 19:58:08.595034 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:08.594985    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9" event={"ID":"ffc78e12-8bf6-4247-b1b7-b3da2c727db5","Type":"ContainerStarted","Data":"e3984be83717e851dc3cef22c33d7ca988581b78cd0558313b1735f70626bc1c"}
Apr 22 19:58:08.597295 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:08.597264    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh" event={"ID":"34b46484-7515-431f-9460-068a93fec7e4","Type":"ContainerStarted","Data":"42d0df6275a9580983b9345a126c9c1439d43740c1a1da1f7c8261c1eec8cdd4"}
Apr 22 19:58:08.597844 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:08.597827    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pmrkj"
Apr 22 19:58:08.598364 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:08.598345    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pmrkj"
Apr 22 19:58:08.613631 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:08.613575    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65cmh" podStartSLOduration=2.764136345 podStartE2EDuration="23.61356087s" podCreationTimestamp="2026-04-22 19:57:45 +0000 UTC" firstStartedPulling="2026-04-22 19:57:46.641847095 +0000 UTC m=+1.741803344" lastFinishedPulling="2026-04-22 19:58:07.491271623 +0000 UTC m=+22.591227869" observedRunningTime="2026-04-22 19:58:08.613458699 +0000 UTC m=+23.713414987" watchObservedRunningTime="2026-04-22 19:58:08.61356087 +0000 UTC m=+23.713517135"
Apr 22 19:58:10.369465 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:10.369249    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:58:10.369887 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:10.369274    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:58:10.369887 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:10.369585    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkqdc" podUID="7dd397be-1edf-42eb-8b45-a3291c8c3b7f"
Apr 22 19:58:10.369887 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:10.369622    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pq8s7" podUID="7f4a64a7-a0cc-426a-9469-5cb2654b3b1d"
Apr 22 19:58:11.605060 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:11.605018    2576 generic.go:358] "Generic (PLEG): container finished" podID="dc0bc97c-7cea-48d0-a254-fb23e782e797" containerID="70fcbe2dec630a1efde3d3408783bbab6642eaa8829f6f9472698eb9b1089029" exitCode=0
Apr 22 19:58:11.605968 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:11.605096    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q25vj" event={"ID":"dc0bc97c-7cea-48d0-a254-fb23e782e797","Type":"ContainerDied","Data":"70fcbe2dec630a1efde3d3408783bbab6642eaa8829f6f9472698eb9b1089029"}
Apr 22 19:58:11.608366 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:11.608340    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9" event={"ID":"ffc78e12-8bf6-4247-b1b7-b3da2c727db5","Type":"ContainerStarted","Data":"26f82cc749cb459d09155c9368d3cab2c4bd4323465c78a724d72f4e4ac51316"}
Apr 22 19:58:11.608695 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:11.608647    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:58:11.608695 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:11.608677    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:58:11.623550 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:11.623529    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:58:11.623715 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:11.623696    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:58:11.645011 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:11.644973    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9" podStartSLOduration=8.104580566 podStartE2EDuration="26.644962086s" podCreationTimestamp="2026-04-22 19:57:45 +0000 UTC" firstStartedPulling="2026-04-22 19:57:46.618325237 +0000 UTC m=+1.718281480" lastFinishedPulling="2026-04-22 19:58:05.158706743 +0000 UTC m=+20.258663000" observedRunningTime="2026-04-22 19:58:11.644661269 +0000 UTC m=+26.744617535" watchObservedRunningTime="2026-04-22 19:58:11.644962086 +0000 UTC m=+26.744918351"
Apr 22 19:58:12.369509 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:12.369323    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:58:12.369636 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:12.369360    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:58:12.369636 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:12.369599    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pq8s7" podUID="7f4a64a7-a0cc-426a-9469-5cb2654b3b1d"
Apr 22 19:58:12.369718 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:12.369682    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkqdc" podUID="7dd397be-1edf-42eb-8b45-a3291c8c3b7f"
Apr 22 19:58:12.593687 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:12.593616    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pq8s7"]
Apr 22 19:58:12.596430 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:12.596406    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lkqdc"]
Apr 22 19:58:12.612137 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:12.612106    2576 generic.go:358] "Generic (PLEG): container finished" podID="dc0bc97c-7cea-48d0-a254-fb23e782e797" containerID="d1350a100b5e32ad65d58280ddc164d2c39c60afaca16878056604d96c8f1b92" exitCode=0
Apr 22 19:58:12.612496 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:12.612191    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q25vj" event={"ID":"dc0bc97c-7cea-48d0-a254-fb23e782e797","Type":"ContainerDied","Data":"d1350a100b5e32ad65d58280ddc164d2c39c60afaca16878056604d96c8f1b92"}
Apr 22 19:58:12.612550 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:12.612507    2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Apr 22 19:58:12.612550 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:12.612527    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:58:12.613016 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:12.612995    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:58:12.613016 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:12.612995    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pq8s7" podUID="7f4a64a7-a0cc-426a-9469-5cb2654b3b1d"
Apr 22 19:58:12.613131 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:12.613086    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkqdc" podUID="7dd397be-1edf-42eb-8b45-a3291c8c3b7f"
Apr 22 19:58:13.254614 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:13.254535    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:58:13.616289 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:13.616196    2576 generic.go:358] "Generic (PLEG): container finished" podID="dc0bc97c-7cea-48d0-a254-fb23e782e797" containerID="c1d58a158771ad958010ced0c0535db5de3c9cf2946b894f0da4a145311d85b6" exitCode=0
Apr 22 19:58:13.616682 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:13.616277    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q25vj" event={"ID":"dc0bc97c-7cea-48d0-a254-fb23e782e797","Type":"ContainerDied","Data":"c1d58a158771ad958010ced0c0535db5de3c9cf2946b894f0da4a145311d85b6"}
Apr 22 19:58:14.369469 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:14.369435    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:58:14.369641 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:14.369491    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:58:14.369641 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:14.369586    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkqdc" podUID="7dd397be-1edf-42eb-8b45-a3291c8c3b7f"
Apr 22 19:58:14.369751 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:14.369723    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pq8s7" podUID="7f4a64a7-a0cc-426a-9469-5cb2654b3b1d"
Apr 22 19:58:16.369533 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:16.369499    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:58:16.369533 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:16.369527    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:58:16.370019 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:16.369641    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkqdc" podUID="7dd397be-1edf-42eb-8b45-a3291c8c3b7f"
Apr 22 19:58:16.370019 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:16.369793    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pq8s7" podUID="7f4a64a7-a0cc-426a-9469-5cb2654b3b1d"
Apr 22 19:58:18.126499 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.126444    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs\") pod \"network-metrics-daemon-lkqdc\" (UID: \"7dd397be-1edf-42eb-8b45-a3291c8c3b7f\") " pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:58:18.126982 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:18.126618    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:58:18.126982 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:18.126702    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs podName:7dd397be-1edf-42eb-8b45-a3291c8c3b7f nodeName:}" failed. No retries permitted until 2026-04-22 19:58:50.12667987 +0000 UTC m=+65.226636117 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs") pod "network-metrics-daemon-lkqdc" (UID: "7dd397be-1edf-42eb-8b45-a3291c8c3b7f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:58:18.227461 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.227421    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8tfnx\" (UniqueName: \"kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx\") pod \"network-check-target-pq8s7\" (UID: \"7f4a64a7-a0cc-426a-9469-5cb2654b3b1d\") " pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:58:18.227639 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:18.227566    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 22 19:58:18.227639 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:18.227582    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 22 19:58:18.227639 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:18.227612    2576 projected.go:194] Error preparing data for projected volume kube-api-access-8tfnx for pod openshift-network-diagnostics/network-check-target-pq8s7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:58:18.227805 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:18.227675    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx podName:7f4a64a7-a0cc-426a-9469-5cb2654b3b1d nodeName:}" failed. No retries permitted until 2026-04-22 19:58:50.227660289 +0000 UTC m=+65.327616533 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8tfnx" (UniqueName: "kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx") pod "network-check-target-pq8s7" (UID: "7f4a64a7-a0cc-426a-9469-5cb2654b3b1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:58:18.233267 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.233241    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-245.ec2.internal" event="NodeReady"
Apr 22 19:58:18.233398 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.233363    2576 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
Apr 22 19:58:18.280835 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.280803    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6hc62"]
Apr 22 19:58:18.319408 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.319367    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gdmz7"]
Apr 22 19:58:18.319573 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.319530    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6hc62"
Apr 22 19:58:18.324614 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.324431    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Apr 22 19:58:18.324790 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.324439    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Apr 22 19:58:18.324856 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.324541    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wrr4x\""
Apr 22 19:58:18.333819 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.333799    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gdmz7"]
Apr 22 19:58:18.333925 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.333823    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6hc62"]
Apr 22 19:58:18.333983 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.333935    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gdmz7"
Apr 22 19:58:18.337727 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.337700    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Apr 22 19:58:18.337836 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.337806    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Apr 22 19:58:18.337836 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.337823    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2xv8f\""
Apr 22 19:58:18.337836 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.337828    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Apr 22 19:58:18.369901 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.369872    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:58:18.370032 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.369904    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:58:18.374232 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.374212    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Apr 22 19:58:18.374361 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.374266    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gl2tf\""
Apr 22 19:58:18.374361 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.374267    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Apr 22 19:58:18.374751 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.374734    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Apr 22 19:58:18.374868 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.374850    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pxnlc\""
Apr 22 19:58:18.428312 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.428282    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/459bfb9a-58fc-428e-80a8-2813ad5247b8-cert\") pod \"ingress-canary-gdmz7\" (UID: \"459bfb9a-58fc-428e-80a8-2813ad5247b8\") " pod="openshift-ingress-canary/ingress-canary-gdmz7"
Apr 22 19:58:18.428472 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.428338    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc8gq\" (UniqueName: \"kubernetes.io/projected/459bfb9a-58fc-428e-80a8-2813ad5247b8-kube-api-access-vc8gq\") pod \"ingress-canary-gdmz7\" (UID: \"459bfb9a-58fc-428e-80a8-2813ad5247b8\") " pod="openshift-ingress-canary/ingress-canary-gdmz7"
Apr 22 19:58:18.428472 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.428370    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8daccaf9-72fb-4f6d-b1b1-245798df3452-config-volume\") pod \"dns-default-6hc62\" (UID: \"8daccaf9-72fb-4f6d-b1b1-245798df3452\") " pod="openshift-dns/dns-default-6hc62"
Apr 22 19:58:18.428472 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.428386    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8daccaf9-72fb-4f6d-b1b1-245798df3452-tmp-dir\") pod \"dns-default-6hc62\" (UID: \"8daccaf9-72fb-4f6d-b1b1-245798df3452\") " pod="openshift-dns/dns-default-6hc62"
Apr 22 19:58:18.428590 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.428486    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8daccaf9-72fb-4f6d-b1b1-245798df3452-metrics-tls\") pod \"dns-default-6hc62\" (UID: \"8daccaf9-72fb-4f6d-b1b1-245798df3452\") " pod="openshift-dns/dns-default-6hc62"
Apr 22 19:58:18.428590 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.428502    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dzzk\" (UniqueName: \"kubernetes.io/projected/8daccaf9-72fb-4f6d-b1b1-245798df3452-kube-api-access-8dzzk\") pod \"dns-default-6hc62\" (UID: \"8daccaf9-72fb-4f6d-b1b1-245798df3452\") " pod="openshift-dns/dns-default-6hc62"
Apr 22 19:58:18.529446 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.529397    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8daccaf9-72fb-4f6d-b1b1-245798df3452-metrics-tls\") pod \"dns-default-6hc62\" (UID: \"8daccaf9-72fb-4f6d-b1b1-245798df3452\") " pod="openshift-dns/dns-default-6hc62"
Apr 22 19:58:18.529446 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.529446    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8dzzk\" (UniqueName: \"kubernetes.io/projected/8daccaf9-72fb-4f6d-b1b1-245798df3452-kube-api-access-8dzzk\") pod \"dns-default-6hc62\" (UID: \"8daccaf9-72fb-4f6d-b1b1-245798df3452\") " pod="openshift-dns/dns-default-6hc62"
Apr 22 19:58:18.529667 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.529501    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/459bfb9a-58fc-428e-80a8-2813ad5247b8-cert\") pod \"ingress-canary-gdmz7\" (UID: \"459bfb9a-58fc-428e-80a8-2813ad5247b8\") " pod="openshift-ingress-canary/ingress-canary-gdmz7"
Apr 22 19:58:18.529667 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.529562    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vc8gq\" (UniqueName: \"kubernetes.io/projected/459bfb9a-58fc-428e-80a8-2813ad5247b8-kube-api-access-vc8gq\") pod \"ingress-canary-gdmz7\" (UID: \"459bfb9a-58fc-428e-80a8-2813ad5247b8\") " pod="openshift-ingress-canary/ingress-canary-gdmz7"
Apr 22 19:58:18.529667 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.529600    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8daccaf9-72fb-4f6d-b1b1-245798df3452-config-volume\") pod \"dns-default-6hc62\" (UID: \"8daccaf9-72fb-4f6d-b1b1-245798df3452\") " pod="openshift-dns/dns-default-6hc62"
Apr 22 19:58:18.529667 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.529618    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8daccaf9-72fb-4f6d-b1b1-245798df3452-tmp-dir\") pod \"dns-default-6hc62\" (UID: \"8daccaf9-72fb-4f6d-b1b1-245798df3452\") " pod="openshift-dns/dns-default-6hc62"
Apr 22 19:58:18.529667 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:18.529567    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 22 19:58:18.529856 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:18.529695    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8daccaf9-72fb-4f6d-b1b1-245798df3452-metrics-tls podName:8daccaf9-72fb-4f6d-b1b1-245798df3452 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:19.029673551 +0000 UTC m=+34.129629793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8daccaf9-72fb-4f6d-b1b1-245798df3452-metrics-tls") pod "dns-default-6hc62" (UID: "8daccaf9-72fb-4f6d-b1b1-245798df3452") : secret "dns-default-metrics-tls" not found
Apr 22 19:58:18.529953 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.529937    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8daccaf9-72fb-4f6d-b1b1-245798df3452-tmp-dir\") pod \"dns-default-6hc62\" (UID: \"8daccaf9-72fb-4f6d-b1b1-245798df3452\") " pod="openshift-dns/dns-default-6hc62"
Apr 22 19:58:18.529992 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:18.529623    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 22 19:58:18.530037 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:18.529998    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/459bfb9a-58fc-428e-80a8-2813ad5247b8-cert podName:459bfb9a-58fc-428e-80a8-2813ad5247b8 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:19.029986581 +0000 UTC m=+34.129942827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/459bfb9a-58fc-428e-80a8-2813ad5247b8-cert") pod "ingress-canary-gdmz7" (UID: "459bfb9a-58fc-428e-80a8-2813ad5247b8") : secret "canary-serving-cert" not found
Apr 22 19:58:18.530250 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.530232    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8daccaf9-72fb-4f6d-b1b1-245798df3452-config-volume\") pod \"dns-default-6hc62\" (UID: \"8daccaf9-72fb-4f6d-b1b1-245798df3452\") " pod="openshift-dns/dns-default-6hc62"
Apr 22 19:58:18.539963 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.539940    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dzzk\" (UniqueName: \"kubernetes.io/projected/8daccaf9-72fb-4f6d-b1b1-245798df3452-kube-api-access-8dzzk\") pod \"dns-default-6hc62\" (UID: \"8daccaf9-72fb-4f6d-b1b1-245798df3452\") " pod="openshift-dns/dns-default-6hc62"
Apr 22 19:58:18.540166 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:18.540124    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc8gq\" (UniqueName: \"kubernetes.io/projected/459bfb9a-58fc-428e-80a8-2813ad5247b8-kube-api-access-vc8gq\") pod \"ingress-canary-gdmz7\" (UID: \"459bfb9a-58fc-428e-80a8-2813ad5247b8\") " pod="openshift-ingress-canary/ingress-canary-gdmz7"
Apr 22 19:58:19.033352 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:19.033316    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/459bfb9a-58fc-428e-80a8-2813ad5247b8-cert\") pod \"ingress-canary-gdmz7\" (UID: \"459bfb9a-58fc-428e-80a8-2813ad5247b8\") " pod="openshift-ingress-canary/ingress-canary-gdmz7"
Apr 22 19:58:19.033550 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:19.033404    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8daccaf9-72fb-4f6d-b1b1-245798df3452-metrics-tls\") pod \"dns-default-6hc62\" (UID: \"8daccaf9-72fb-4f6d-b1b1-245798df3452\") " pod="openshift-dns/dns-default-6hc62"
Apr 22 19:58:19.033550 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:19.033483    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 22 19:58:19.033550 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:19.033512    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 22 19:58:19.033655 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:19.033565    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8daccaf9-72fb-4f6d-b1b1-245798df3452-metrics-tls podName:8daccaf9-72fb-4f6d-b1b1-245798df3452 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:20.033550234 +0000 UTC m=+35.133506480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8daccaf9-72fb-4f6d-b1b1-245798df3452-metrics-tls") pod "dns-default-6hc62" (UID: "8daccaf9-72fb-4f6d-b1b1-245798df3452") : secret "dns-default-metrics-tls" not found
Apr 22 19:58:19.033655 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:19.033583    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/459bfb9a-58fc-428e-80a8-2813ad5247b8-cert podName:459bfb9a-58fc-428e-80a8-2813ad5247b8 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:20.033573725 +0000 UTC m=+35.133529971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/459bfb9a-58fc-428e-80a8-2813ad5247b8-cert") pod "ingress-canary-gdmz7" (UID: "459bfb9a-58fc-428e-80a8-2813ad5247b8") : secret "canary-serving-cert" not found
Apr 22 19:58:20.040371 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:20.040332    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8daccaf9-72fb-4f6d-b1b1-245798df3452-metrics-tls\") pod \"dns-default-6hc62\" (UID: \"8daccaf9-72fb-4f6d-b1b1-245798df3452\") " pod="openshift-dns/dns-default-6hc62"
Apr 22 19:58:20.040771 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:20.040385    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/459bfb9a-58fc-428e-80a8-2813ad5247b8-cert\") pod \"ingress-canary-gdmz7\" (UID: \"459bfb9a-58fc-428e-80a8-2813ad5247b8\") " pod="openshift-ingress-canary/ingress-canary-gdmz7"
Apr 22 19:58:20.040771 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:20.040483    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 22 19:58:20.040771 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:20.040483    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 22 19:58:20.040771 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:20.040545    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/459bfb9a-58fc-428e-80a8-2813ad5247b8-cert podName:459bfb9a-58fc-428e-80a8-2813ad5247b8 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:22.040530822 +0000 UTC m=+37.140487068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/459bfb9a-58fc-428e-80a8-2813ad5247b8-cert") pod "ingress-canary-gdmz7" (UID: "459bfb9a-58fc-428e-80a8-2813ad5247b8") : secret "canary-serving-cert" not found
Apr 22 19:58:20.040771 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:20.040561    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8daccaf9-72fb-4f6d-b1b1-245798df3452-metrics-tls podName:8daccaf9-72fb-4f6d-b1b1-245798df3452 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:22.040554109 +0000 UTC m=+37.140510352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8daccaf9-72fb-4f6d-b1b1-245798df3452-metrics-tls") pod "dns-default-6hc62" (UID: "8daccaf9-72fb-4f6d-b1b1-245798df3452") : secret "dns-default-metrics-tls" not found
Apr 22 19:58:20.632696 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:20.632665    2576 generic.go:358] "Generic (PLEG): container finished" podID="dc0bc97c-7cea-48d0-a254-fb23e782e797" containerID="11988be158ab4b52fb9c95156aa2bea71f8546525e0cd28f02ae70fc28ea2dc9" exitCode=0
Apr 22 19:58:20.632946 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:20.632722    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q25vj" event={"ID":"dc0bc97c-7cea-48d0-a254-fb23e782e797","Type":"ContainerDied","Data":"11988be158ab4b52fb9c95156aa2bea71f8546525e0cd28f02ae70fc28ea2dc9"}
Apr 22 19:58:21.637368 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:21.637333    2576 generic.go:358] "Generic (PLEG): container finished" podID="dc0bc97c-7cea-48d0-a254-fb23e782e797" containerID="54f0f554bb6c55e4f7795542248585bba24b92111f373cc72380b43a987ccf6b" exitCode=0
Apr 22 19:58:21.637832 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:21.637398    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q25vj" event={"ID":"dc0bc97c-7cea-48d0-a254-fb23e782e797","Type":"ContainerDied","Data":"54f0f554bb6c55e4f7795542248585bba24b92111f373cc72380b43a987ccf6b"}
Apr 22 19:58:22.054952 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:22.054873    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/459bfb9a-58fc-428e-80a8-2813ad5247b8-cert\") pod \"ingress-canary-gdmz7\" (UID: \"459bfb9a-58fc-428e-80a8-2813ad5247b8\") " pod="openshift-ingress-canary/ingress-canary-gdmz7"
Apr 22 19:58:22.054952 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:22.054951    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8daccaf9-72fb-4f6d-b1b1-245798df3452-metrics-tls\") pod \"dns-default-6hc62\" (UID: \"8daccaf9-72fb-4f6d-b1b1-245798df3452\") " pod="openshift-dns/dns-default-6hc62"
Apr 22 19:58:22.055125 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:22.055037    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 22 19:58:22.055125 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:22.055040    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 22 19:58:22.055125 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:22.055091    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8daccaf9-72fb-4f6d-b1b1-245798df3452-metrics-tls podName:8daccaf9-72fb-4f6d-b1b1-245798df3452 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:26.055078246 +0000 UTC m=+41.155034489 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8daccaf9-72fb-4f6d-b1b1-245798df3452-metrics-tls") pod "dns-default-6hc62" (UID: "8daccaf9-72fb-4f6d-b1b1-245798df3452") : secret "dns-default-metrics-tls" not found
Apr 22 19:58:22.055125 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:22.055104    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/459bfb9a-58fc-428e-80a8-2813ad5247b8-cert podName:459bfb9a-58fc-428e-80a8-2813ad5247b8 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:26.055098563 +0000 UTC m=+41.155054806 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/459bfb9a-58fc-428e-80a8-2813ad5247b8-cert") pod "ingress-canary-gdmz7" (UID: "459bfb9a-58fc-428e-80a8-2813ad5247b8") : secret "canary-serving-cert" not found
Apr 22 19:58:22.642069 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:22.642030    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q25vj" event={"ID":"dc0bc97c-7cea-48d0-a254-fb23e782e797","Type":"ContainerStarted","Data":"9c89418f9f45a9995527e844e9e63a244a0d9c6a0f76b9d06a8ce52b8635a4ff"}
Apr 22 19:58:22.665522 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:22.665467    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q25vj" podStartSLOduration=4.697461324 podStartE2EDuration="37.66545188s" podCreationTimestamp="2026-04-22 19:57:45 +0000 UTC" firstStartedPulling="2026-04-22 19:57:46.629338296 +0000 UTC m=+1.729294540" lastFinishedPulling="2026-04-22 19:58:19.597328854 +0000 UTC m=+34.697285096" observedRunningTime="2026-04-22 19:58:22.663808121 +0000 UTC m=+37.763764386" watchObservedRunningTime="2026-04-22 19:58:22.66545188 +0000 UTC m=+37.765408144"
Apr 22 19:58:26.083136 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:26.083097    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8daccaf9-72fb-4f6d-b1b1-245798df3452-metrics-tls\") pod \"dns-default-6hc62\" (UID: \"8daccaf9-72fb-4f6d-b1b1-245798df3452\") " pod="openshift-dns/dns-default-6hc62"
Apr 22 19:58:26.083136 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:26.083144    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/459bfb9a-58fc-428e-80a8-2813ad5247b8-cert\") pod \"ingress-canary-gdmz7\" (UID: \"459bfb9a-58fc-428e-80a8-2813ad5247b8\") " pod="openshift-ingress-canary/ingress-canary-gdmz7"
Apr 22 19:58:26.083610 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:26.083248    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 22 19:58:26.083610 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:26.083251    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 22 19:58:26.083610 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:26.083299    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/459bfb9a-58fc-428e-80a8-2813ad5247b8-cert podName:459bfb9a-58fc-428e-80a8-2813ad5247b8 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:34.083285528 +0000 UTC m=+49.183241771 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/459bfb9a-58fc-428e-80a8-2813ad5247b8-cert") pod "ingress-canary-gdmz7" (UID: "459bfb9a-58fc-428e-80a8-2813ad5247b8") : secret "canary-serving-cert" not found
Apr 22 19:58:26.083610 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:26.083312    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8daccaf9-72fb-4f6d-b1b1-245798df3452-metrics-tls podName:8daccaf9-72fb-4f6d-b1b1-245798df3452 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:34.083306442 +0000 UTC m=+49.183262686 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8daccaf9-72fb-4f6d-b1b1-245798df3452-metrics-tls") pod "dns-default-6hc62" (UID: "8daccaf9-72fb-4f6d-b1b1-245798df3452") : secret "dns-default-metrics-tls" not found
Apr 22 19:58:34.134431 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:34.134390    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/459bfb9a-58fc-428e-80a8-2813ad5247b8-cert\") pod \"ingress-canary-gdmz7\" (UID: \"459bfb9a-58fc-428e-80a8-2813ad5247b8\") " pod="openshift-ingress-canary/ingress-canary-gdmz7"
Apr 22 19:58:34.134880 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:34.134462    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8daccaf9-72fb-4f6d-b1b1-245798df3452-metrics-tls\") pod \"dns-default-6hc62\" (UID: \"8daccaf9-72fb-4f6d-b1b1-245798df3452\") " pod="openshift-dns/dns-default-6hc62"
Apr 22 19:58:34.134880 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:34.134547    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 22 19:58:34.134880 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:34.134548    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 22 19:58:34.134880 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:34.134627    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8daccaf9-72fb-4f6d-b1b1-245798df3452-metrics-tls podName:8daccaf9-72fb-4f6d-b1b1-245798df3452 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:50.134591717 +0000 UTC m=+65.234547962 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8daccaf9-72fb-4f6d-b1b1-245798df3452-metrics-tls") pod "dns-default-6hc62" (UID: "8daccaf9-72fb-4f6d-b1b1-245798df3452") : secret "dns-default-metrics-tls" not found
Apr 22 19:58:34.134880 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:58:34.134646    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/459bfb9a-58fc-428e-80a8-2813ad5247b8-cert podName:459bfb9a-58fc-428e-80a8-2813ad5247b8 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:50.1346373 +0000 UTC m=+65.234593543 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/459bfb9a-58fc-428e-80a8-2813ad5247b8-cert") pod "ingress-canary-gdmz7" (UID: "459bfb9a-58fc-428e-80a8-2813ad5247b8") : secret "canary-serving-cert" not found
Apr 22 19:58:35.840685 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:35.840654    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rmblf_f12eca63-f2df-4781-97dd-7968beea64ba/dns-node-resolver/0.log"
Apr 22 19:58:36.803119 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:36.803083    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-865cb79987-88mw4"]
Apr 22 19:58:36.805871 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:36.805851    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-865cb79987-88mw4"
Apr 22 19:58:36.808377 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:36.808352    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Apr 22 19:58:36.808482 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:36.808383    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Apr 22 19:58:36.809028 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:36.809014    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Apr 22 19:58:36.809113 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:36.809099    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-l9tms\""
Apr 22 19:58:36.809179 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:36.809109    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Apr 22 19:58:36.813440 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:36.813407    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-865cb79987-88mw4"]
Apr 22 19:58:36.840273 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:36.840242    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tn59d_671f3faa-9804-447f-acb1-14a0c42815dd/node-ca/0.log"
Apr 22 19:58:36.954189 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:36.954140    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ccf538b4-6cf3-4673-b55c-eb3e02f925cd-signing-cabundle\") pod \"service-ca-865cb79987-88mw4\" (UID: \"ccf538b4-6cf3-4673-b55c-eb3e02f925cd\") " pod="openshift-service-ca/service-ca-865cb79987-88mw4"
Apr 22 19:58:36.954548 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:36.954226    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tkl9\" (UniqueName: \"kubernetes.io/projected/ccf538b4-6cf3-4673-b55c-eb3e02f925cd-kube-api-access-4tkl9\") pod \"service-ca-865cb79987-88mw4\" (UID: \"ccf538b4-6cf3-4673-b55c-eb3e02f925cd\") " pod="openshift-service-ca/service-ca-865cb79987-88mw4"
Apr 22 19:58:36.954548 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:36.954260    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ccf538b4-6cf3-4673-b55c-eb3e02f925cd-signing-key\") pod \"service-ca-865cb79987-88mw4\" (UID: \"ccf538b4-6cf3-4673-b55c-eb3e02f925cd\") " pod="openshift-service-ca/service-ca-865cb79987-88mw4"
Apr 22 19:58:37.054852 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:37.054751    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ccf538b4-6cf3-4673-b55c-eb3e02f925cd-signing-cabundle\") pod \"service-ca-865cb79987-88mw4\" (UID: \"ccf538b4-6cf3-4673-b55c-eb3e02f925cd\") " pod="openshift-service-ca/service-ca-865cb79987-88mw4"
Apr 22 19:58:37.054852 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:37.054847    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4tkl9\" (UniqueName: \"kubernetes.io/projected/ccf538b4-6cf3-4673-b55c-eb3e02f925cd-kube-api-access-4tkl9\") pod \"service-ca-865cb79987-88mw4\" (UID: \"ccf538b4-6cf3-4673-b55c-eb3e02f925cd\") " pod="openshift-service-ca/service-ca-865cb79987-88mw4"
Apr 22 19:58:37.055072 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:37.054906    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ccf538b4-6cf3-4673-b55c-eb3e02f925cd-signing-key\") pod \"service-ca-865cb79987-88mw4\" (UID: \"ccf538b4-6cf3-4673-b55c-eb3e02f925cd\") " pod="openshift-service-ca/service-ca-865cb79987-88mw4"
Apr 22 19:58:37.055444 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:37.055422    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ccf538b4-6cf3-4673-b55c-eb3e02f925cd-signing-cabundle\") pod \"service-ca-865cb79987-88mw4\" (UID: \"ccf538b4-6cf3-4673-b55c-eb3e02f925cd\") " pod="openshift-service-ca/service-ca-865cb79987-88mw4"
Apr 22 19:58:37.057597 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:37.057576    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ccf538b4-6cf3-4673-b55c-eb3e02f925cd-signing-key\") pod \"service-ca-865cb79987-88mw4\" (UID: \"ccf538b4-6cf3-4673-b55c-eb3e02f925cd\") " pod="openshift-service-ca/service-ca-865cb79987-88mw4"
Apr 22 19:58:37.063252 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:37.063230    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tkl9\" (UniqueName: \"kubernetes.io/projected/ccf538b4-6cf3-4673-b55c-eb3e02f925cd-kube-api-access-4tkl9\") pod \"service-ca-865cb79987-88mw4\" (UID: \"ccf538b4-6cf3-4673-b55c-eb3e02f925cd\") " pod="openshift-service-ca/service-ca-865cb79987-88mw4"
Apr 22 19:58:37.115619 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:37.115591    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-865cb79987-88mw4"
Apr 22 19:58:37.236644 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:37.236615    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-865cb79987-88mw4"]
Apr 22 19:58:37.241185 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:58:37.241133    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf538b4_6cf3_4673_b55c_eb3e02f925cd.slice/crio-578b7a27b7fcb5698b6d2ab394efa2e1bc19ac65948dfc3576946254ca4291de WatchSource:0}: Error finding container 578b7a27b7fcb5698b6d2ab394efa2e1bc19ac65948dfc3576946254ca4291de: Status 404 returned error can't find the container with id 578b7a27b7fcb5698b6d2ab394efa2e1bc19ac65948dfc3576946254ca4291de
Apr 22 19:58:37.671868 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:37.671829    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-865cb79987-88mw4" event={"ID":"ccf538b4-6cf3-4673-b55c-eb3e02f925cd","Type":"ContainerStarted","Data":"578b7a27b7fcb5698b6d2ab394efa2e1bc19ac65948dfc3576946254ca4291de"}
Apr 22 19:58:39.676678 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:39.676640    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-865cb79987-88mw4" event={"ID":"ccf538b4-6cf3-4673-b55c-eb3e02f925cd","Type":"ContainerStarted","Data":"4e72c57763aee6bfbc6513567d829bf179cbf4f3cc2a0e10623cfadd2a3c021a"}
Apr 22 19:58:39.691632 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:39.691587    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-865cb79987-88mw4" podStartSLOduration=2.084801591 podStartE2EDuration="3.691572321s" podCreationTimestamp="2026-04-22 19:58:36 +0000 UTC" firstStartedPulling="2026-04-22 19:58:37.242980534 +0000 UTC m=+52.342936779" lastFinishedPulling="2026-04-22 19:58:38.849751263 +0000 UTC m=+53.949707509" observedRunningTime="2026-04-22 19:58:39.691006308 +0000 UTC m=+54.790962698" watchObservedRunningTime="2026-04-22 19:58:39.691572321 +0000 UTC m=+54.791528620"
Apr 22 19:58:44.657256 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:44.657224    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rbkp9"
Apr 22 19:58:50.148876 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.148835    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/459bfb9a-58fc-428e-80a8-2813ad5247b8-cert\") pod \"ingress-canary-gdmz7\" (UID: \"459bfb9a-58fc-428e-80a8-2813ad5247b8\") " pod="openshift-ingress-canary/ingress-canary-gdmz7"
Apr 22 19:58:50.149358 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.148920    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs\") pod \"network-metrics-daemon-lkqdc\" (UID: \"7dd397be-1edf-42eb-8b45-a3291c8c3b7f\") " pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:58:50.149358 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.148954    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8daccaf9-72fb-4f6d-b1b1-245798df3452-metrics-tls\") pod \"dns-default-6hc62\" (UID: \"8daccaf9-72fb-4f6d-b1b1-245798df3452\") " pod="openshift-dns/dns-default-6hc62"
Apr 22 19:58:50.151289 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.151261    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Apr 22 19:58:50.151559 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.151540    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8daccaf9-72fb-4f6d-b1b1-245798df3452-metrics-tls\") pod \"dns-default-6hc62\" (UID: \"8daccaf9-72fb-4f6d-b1b1-245798df3452\") " pod="openshift-dns/dns-default-6hc62"
Apr 22 19:58:50.151771 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.151750    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/459bfb9a-58fc-428e-80a8-2813ad5247b8-cert\") pod \"ingress-canary-gdmz7\" (UID: \"459bfb9a-58fc-428e-80a8-2813ad5247b8\") " pod="openshift-ingress-canary/ingress-canary-gdmz7"
Apr 22 19:58:50.162281 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.162248    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dd397be-1edf-42eb-8b45-a3291c8c3b7f-metrics-certs\") pod \"network-metrics-daemon-lkqdc\" (UID: \"7dd397be-1edf-42eb-8b45-a3291c8c3b7f\") " pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:58:50.189067 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.189036    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gl2tf\""
Apr 22 19:58:50.196995 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.196977    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkqdc"
Apr 22 19:58:50.250212 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.250119    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8tfnx\" (UniqueName: \"kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx\") pod \"network-check-target-pq8s7\" (UID: \"7f4a64a7-a0cc-426a-9469-5cb2654b3b1d\") " pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:58:50.252553 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.252523    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Apr 22 19:58:50.263370 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.263344    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Apr 22 19:58:50.274870 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.274842    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tfnx\" (UniqueName: \"kubernetes.io/projected/7f4a64a7-a0cc-426a-9469-5cb2654b3b1d-kube-api-access-8tfnx\") pod \"network-check-target-pq8s7\" (UID: \"7f4a64a7-a0cc-426a-9469-5cb2654b3b1d\") " pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:58:50.315094 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.315060    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lkqdc"]
Apr 22 19:58:50.318352 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:58:50.318325    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd397be_1edf_42eb_8b45_a3291c8c3b7f.slice/crio-f0dabb67bfdc9f6046cde6de2982d8693d028789edc9b30d475d768dd4459e32 WatchSource:0}: Error finding container f0dabb67bfdc9f6046cde6de2982d8693d028789edc9b30d475d768dd4459e32: Status 404 returned error can't find the container with id f0dabb67bfdc9f6046cde6de2982d8693d028789edc9b30d475d768dd4459e32
Apr 22 19:58:50.433516 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.433429    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wrr4x\""
Apr 22 19:58:50.441447 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.441427    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6hc62"
Apr 22 19:58:50.448189 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.448107    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2xv8f\""
Apr 22 19:58:50.456979 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.456954    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gdmz7"
Apr 22 19:58:50.482993 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.482965    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pxnlc\""
Apr 22 19:58:50.491960 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.491688    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:58:50.571443 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.571415    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6hc62"]
Apr 22 19:58:50.575644 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:58:50.575599    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8daccaf9_72fb_4f6d_b1b1_245798df3452.slice/crio-b8f4ac8cd8ddf94ee1bce56e392ce4101fc86622f44b663a406899fd0627fc3f WatchSource:0}: Error finding container b8f4ac8cd8ddf94ee1bce56e392ce4101fc86622f44b663a406899fd0627fc3f: Status 404 returned error can't find the container with id b8f4ac8cd8ddf94ee1bce56e392ce4101fc86622f44b663a406899fd0627fc3f
Apr 22 19:58:50.586970 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.586944    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gdmz7"]
Apr 22 19:58:50.590088 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:58:50.590039    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459bfb9a_58fc_428e_80a8_2813ad5247b8.slice/crio-95679aef5631153822b192176fd1acf53caadec5fc4327d8bb960f32da2e93c9 WatchSource:0}: Error finding container 95679aef5631153822b192176fd1acf53caadec5fc4327d8bb960f32da2e93c9: Status 404 returned error can't find the container with id 95679aef5631153822b192176fd1acf53caadec5fc4327d8bb960f32da2e93c9
Apr 22 19:58:50.666351 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.666321    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pq8s7"]
Apr 22 19:58:50.669364 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:58:50.669337    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4a64a7_a0cc_426a_9469_5cb2654b3b1d.slice/crio-3d66bcd4b636edde6fa0a76245c148cdcc8a873c66c90c2ce00d470a9cf28d31 WatchSource:0}: Error finding container 3d66bcd4b636edde6fa0a76245c148cdcc8a873c66c90c2ce00d470a9cf28d31: Status 404 returned error can't find the container with id 3d66bcd4b636edde6fa0a76245c148cdcc8a873c66c90c2ce00d470a9cf28d31
Apr 22 19:58:50.699224 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.699134    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gdmz7" event={"ID":"459bfb9a-58fc-428e-80a8-2813ad5247b8","Type":"ContainerStarted","Data":"95679aef5631153822b192176fd1acf53caadec5fc4327d8bb960f32da2e93c9"}
Apr 22 19:58:50.700092 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.700067    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lkqdc" event={"ID":"7dd397be-1edf-42eb-8b45-a3291c8c3b7f","Type":"ContainerStarted","Data":"f0dabb67bfdc9f6046cde6de2982d8693d028789edc9b30d475d768dd4459e32"}
Apr 22 19:58:50.701028 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.701010    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pq8s7" event={"ID":"7f4a64a7-a0cc-426a-9469-5cb2654b3b1d","Type":"ContainerStarted","Data":"3d66bcd4b636edde6fa0a76245c148cdcc8a873c66c90c2ce00d470a9cf28d31"}
Apr 22 19:58:50.701908 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:50.701887    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6hc62" event={"ID":"8daccaf9-72fb-4f6d-b1b1-245798df3452","Type":"ContainerStarted","Data":"b8f4ac8cd8ddf94ee1bce56e392ce4101fc86622f44b663a406899fd0627fc3f"}
Apr 22 19:58:53.710476 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:53.710442    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lkqdc" event={"ID":"7dd397be-1edf-42eb-8b45-a3291c8c3b7f","Type":"ContainerStarted","Data":"3e5fa1de53f437c5c40b43adaa79ac19296dbcdf7f0a3cc7d28cf8361a4c9b85"}
Apr 22 19:58:54.714846 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:54.714805    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lkqdc" event={"ID":"7dd397be-1edf-42eb-8b45-a3291c8c3b7f","Type":"ContainerStarted","Data":"aa62999faa1fabb3dcb7241e7dbf27b1854161651e984e250a5232c3bde7c578"}
Apr 22 19:58:54.716164 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:54.716126    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pq8s7" event={"ID":"7f4a64a7-a0cc-426a-9469-5cb2654b3b1d","Type":"ContainerStarted","Data":"c7354dade2c8aa0cf91ea475d5c4e62dd6850d393e97ba508cfa9d3afb9038d0"}
Apr 22 19:58:54.716287 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:54.716266    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:58:54.717620 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:54.717599    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6hc62" event={"ID":"8daccaf9-72fb-4f6d-b1b1-245798df3452","Type":"ContainerStarted","Data":"97664fc869a46f394c940fa16168e2c74064bee3be2c6ce99c6acc6bcd961ff5"}
Apr 22 19:58:54.717698 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:54.717625    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6hc62" event={"ID":"8daccaf9-72fb-4f6d-b1b1-245798df3452","Type":"ContainerStarted","Data":"e945ff5e44087ae0ece06d828683409549b78b90a3dfcb89c600e7b16818d597"}
Apr 22 19:58:54.717762 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:54.717736    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6hc62"
Apr 22 19:58:54.718831 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:54.718809    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gdmz7" event={"ID":"459bfb9a-58fc-428e-80a8-2813ad5247b8","Type":"ContainerStarted","Data":"61d999891f9ae6194c40b5718628a52255071b3b3b60e56c8ed1e7c45998e314"}
Apr 22 19:58:54.735045 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:54.734994    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lkqdc" podStartSLOduration=67.22403219 podStartE2EDuration="1m9.73498008s" podCreationTimestamp="2026-04-22 19:57:45 +0000 UTC" firstStartedPulling="2026-04-22 19:58:50.320543498 +0000 UTC m=+65.420499741" lastFinishedPulling="2026-04-22 19:58:52.831491387 +0000 UTC m=+67.931447631" observedRunningTime="2026-04-22 19:58:54.73473678 +0000 UTC m=+69.834693045" watchObservedRunningTime="2026-04-22 19:58:54.73498008 +0000 UTC m=+69.834936347"
Apr 22 19:58:54.751419 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:54.751364    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-6hc62" podStartSLOduration=34.495501509 podStartE2EDuration="36.751348687s" podCreationTimestamp="2026-04-22 19:58:18 +0000 UTC" firstStartedPulling="2026-04-22 19:58:50.577395199 +0000 UTC m=+65.677351441" lastFinishedPulling="2026-04-22 19:58:52.833242361 +0000 UTC m=+67.933198619" observedRunningTime="2026-04-22 19:58:54.750845116 +0000 UTC m=+69.850801406" watchObservedRunningTime="2026-04-22 19:58:54.751348687 +0000 UTC m=+69.851304955"
Apr 22 19:58:54.778573 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:54.778529    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gdmz7" podStartSLOduration=34.533943434 podStartE2EDuration="36.778515575s" podCreationTimestamp="2026-04-22 19:58:18 +0000 UTC" firstStartedPulling="2026-04-22 19:58:50.593319666 +0000 UTC m=+65.693275909" lastFinishedPulling="2026-04-22 19:58:52.837891793 +0000 UTC m=+67.937848050" observedRunningTime="2026-04-22 19:58:54.7784927 +0000 UTC m=+69.878448964" watchObservedRunningTime="2026-04-22 19:58:54.778515575 +0000 UTC m=+69.878471840"
Apr 22 19:58:54.778856 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:54.778816    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pq8s7" podStartSLOduration=66.549047467 podStartE2EDuration="1m9.778806542s" podCreationTimestamp="2026-04-22 19:57:45 +0000 UTC" firstStartedPulling="2026-04-22 19:58:50.67154077 +0000 UTC m=+65.771497015" lastFinishedPulling="2026-04-22 19:58:53.90129984 +0000 UTC m=+69.001256090" observedRunningTime="2026-04-22 19:58:54.765014392 +0000 UTC m=+69.864970672" watchObservedRunningTime="2026-04-22 19:58:54.778806542 +0000 UTC m=+69.878762807"
Apr 22 19:58:57.017511 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.017482    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4xsbf"]
Apr 22 19:58:57.020580 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.020563    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4xsbf"
Apr 22 19:58:57.022723 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.022696    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Apr 22 19:58:57.022852 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.022831    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Apr 22 19:58:57.022965 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.022949    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Apr 22 19:58:57.023729 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.023712    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t9dg2\""
Apr 22 19:58:57.023779 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.023716    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Apr 22 19:58:57.041202 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.041180    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4xsbf"]
Apr 22 19:58:57.087204 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.087171    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-pp5hw"]
Apr 22 19:58:57.089932 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.089916    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-pp5hw"
Apr 22 19:58:57.092653 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.092626    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-nhwzw\""
Apr 22 19:58:57.092844 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.092831    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Apr 22 19:58:57.094901 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.094880    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6bcc868b7-fjzng"]
Apr 22 19:58:57.097762 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.097746    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6bcc868b7-fjzng"
Apr 22 19:58:57.100383 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.100340    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-68tx2\""
Apr 22 19:58:57.100383 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.100376    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Apr 22 19:58:57.100515 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.100492    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Apr 22 19:58:57.111870 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.111849    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-pp5hw"]
Apr 22 19:58:57.120175 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.120126    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6bcc868b7-fjzng"]
Apr 22 19:58:57.186842 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.186810    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5db49c5564-f22ph"]
Apr 22 19:58:57.189697 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.189681    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.191928 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.191900    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Apr 22 19:58:57.191928 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.191921    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Apr 22 19:58:57.192078 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.192034    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Apr 22 19:58:57.192335 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.192320    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wxhns\""
Apr 22 19:58:57.196514 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.196495    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Apr 22 19:58:57.198490 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.198462    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a4957aa1-9cf4-499e-af91-57083b2288d6-crio-socket\") pod \"insights-runtime-extractor-4xsbf\" (UID: \"a4957aa1-9cf4-499e-af91-57083b2288d6\") " pod="openshift-insights/insights-runtime-extractor-4xsbf"
Apr 22 19:58:57.198626 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.198511    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh8tp\" (UniqueName: \"kubernetes.io/projected/cf9153e8-d813-4e2c-bd0a-1140521bf7df-kube-api-access-nh8tp\") pod \"downloads-6bcc868b7-fjzng\" (UID: \"cf9153e8-d813-4e2c-bd0a-1140521bf7df\") " pod="openshift-console/downloads-6bcc868b7-fjzng"
Apr 22 19:58:57.198626 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.198576    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4957aa1-9cf4-499e-af91-57083b2288d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4xsbf\" (UID: \"a4957aa1-9cf4-499e-af91-57083b2288d6\") " pod="openshift-insights/insights-runtime-extractor-4xsbf"
Apr 22 19:58:57.198626 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.198599    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4957aa1-9cf4-499e-af91-57083b2288d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4xsbf\" (UID: \"a4957aa1-9cf4-499e-af91-57083b2288d6\") " pod="openshift-insights/insights-runtime-extractor-4xsbf"
Apr 22 19:58:57.198759 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.198637    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d8f4ac74-fc98-4d51-8bb7-136b4695a7bb-tls-certificates\") pod \"prometheus-operator-admission-webhook-57cf98b594-pp5hw\" (UID: \"d8f4ac74-fc98-4d51-8bb7-136b4695a7bb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-pp5hw"
Apr 22 19:58:57.198759 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.198664    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpn4b\" (UniqueName: \"kubernetes.io/projected/a4957aa1-9cf4-499e-af91-57083b2288d6-kube-api-access-zpn4b\") pod \"insights-runtime-extractor-4xsbf\" (UID: \"a4957aa1-9cf4-499e-af91-57083b2288d6\") " pod="openshift-insights/insights-runtime-extractor-4xsbf"
Apr 22 19:58:57.198759 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.198681    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4957aa1-9cf4-499e-af91-57083b2288d6-data-volume\") pod \"insights-runtime-extractor-4xsbf\" (UID: \"a4957aa1-9cf4-499e-af91-57083b2288d6\") " pod="openshift-insights/insights-runtime-extractor-4xsbf"
Apr 22 19:58:57.203190 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.203142    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5db49c5564-f22ph"]
Apr 22 19:58:57.299218 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.299122    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4957aa1-9cf4-499e-af91-57083b2288d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4xsbf\" (UID: \"a4957aa1-9cf4-499e-af91-57083b2288d6\") " pod="openshift-insights/insights-runtime-extractor-4xsbf"
Apr 22 19:58:57.299218 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.299174    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4957aa1-9cf4-499e-af91-57083b2288d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4xsbf\" (UID: \"a4957aa1-9cf4-499e-af91-57083b2288d6\") " pod="openshift-insights/insights-runtime-extractor-4xsbf"
Apr 22 19:58:57.299218 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.299201    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f66e796-5123-4e93-bfe8-da6796c18fe9-registry-tls\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.299218 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.299220    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6f66e796-5123-4e93-bfe8-da6796c18fe9-ca-trust-extracted\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.299473 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.299236    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6f66e796-5123-4e93-bfe8-da6796c18fe9-trusted-ca\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.299473 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.299254    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6f66e796-5123-4e93-bfe8-da6796c18fe9-image-registry-private-configuration\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.299473 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.299276    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6f66e796-5123-4e93-bfe8-da6796c18fe9-registry-certificates\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.299473 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.299329    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-255mh\" (UniqueName: \"kubernetes.io/projected/6f66e796-5123-4e93-bfe8-da6796c18fe9-kube-api-access-255mh\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.299473 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.299407    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f66e796-5123-4e93-bfe8-da6796c18fe9-bound-sa-token\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.299473 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.299463    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d8f4ac74-fc98-4d51-8bb7-136b4695a7bb-tls-certificates\") pod \"prometheus-operator-admission-webhook-57cf98b594-pp5hw\" (UID: \"d8f4ac74-fc98-4d51-8bb7-136b4695a7bb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-pp5hw"
Apr 22 19:58:57.299721 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.299485    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zpn4b\" (UniqueName: \"kubernetes.io/projected/a4957aa1-9cf4-499e-af91-57083b2288d6-kube-api-access-zpn4b\") pod \"insights-runtime-extractor-4xsbf\" (UID: \"a4957aa1-9cf4-499e-af91-57083b2288d6\") " pod="openshift-insights/insights-runtime-extractor-4xsbf"
Apr 22 19:58:57.299721 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.299505    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6f66e796-5123-4e93-bfe8-da6796c18fe9-installation-pull-secrets\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.299721 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.299534    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4957aa1-9cf4-499e-af91-57083b2288d6-data-volume\") pod \"insights-runtime-extractor-4xsbf\" (UID: \"a4957aa1-9cf4-499e-af91-57083b2288d6\") " pod="openshift-insights/insights-runtime-extractor-4xsbf"
Apr 22 19:58:57.299721 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.299603    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a4957aa1-9cf4-499e-af91-57083b2288d6-crio-socket\") pod \"insights-runtime-extractor-4xsbf\" (UID: \"a4957aa1-9cf4-499e-af91-57083b2288d6\") " pod="openshift-insights/insights-runtime-extractor-4xsbf"
Apr 22 19:58:57.299721 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.299653    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nh8tp\" (UniqueName: \"kubernetes.io/projected/cf9153e8-d813-4e2c-bd0a-1140521bf7df-kube-api-access-nh8tp\") pod \"downloads-6bcc868b7-fjzng\" (UID: \"cf9153e8-d813-4e2c-bd0a-1140521bf7df\") " pod="openshift-console/downloads-6bcc868b7-fjzng"
Apr 22 19:58:57.299721 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.299695    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a4957aa1-9cf4-499e-af91-57083b2288d6-crio-socket\") pod \"insights-runtime-extractor-4xsbf\" (UID: \"a4957aa1-9cf4-499e-af91-57083b2288d6\") " pod="openshift-insights/insights-runtime-extractor-4xsbf"
Apr 22 19:58:57.299721 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.299719    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4957aa1-9cf4-499e-af91-57083b2288d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4xsbf\" (UID: \"a4957aa1-9cf4-499e-af91-57083b2288d6\") " pod="openshift-insights/insights-runtime-extractor-4xsbf"
Apr 22 19:58:57.300034 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.299802    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4957aa1-9cf4-499e-af91-57083b2288d6-data-volume\") pod \"insights-runtime-extractor-4xsbf\" (UID: \"a4957aa1-9cf4-499e-af91-57083b2288d6\") " pod="openshift-insights/insights-runtime-extractor-4xsbf"
Apr 22 19:58:57.301782 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.301750    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4957aa1-9cf4-499e-af91-57083b2288d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4xsbf\" (UID: \"a4957aa1-9cf4-499e-af91-57083b2288d6\") " pod="openshift-insights/insights-runtime-extractor-4xsbf"
Apr 22 19:58:57.301936 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.301921    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d8f4ac74-fc98-4d51-8bb7-136b4695a7bb-tls-certificates\") pod \"prometheus-operator-admission-webhook-57cf98b594-pp5hw\" (UID: \"d8f4ac74-fc98-4d51-8bb7-136b4695a7bb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-pp5hw"
Apr 22 19:58:57.315536 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.315505    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpn4b\" (UniqueName: \"kubernetes.io/projected/a4957aa1-9cf4-499e-af91-57083b2288d6-kube-api-access-zpn4b\") pod \"insights-runtime-extractor-4xsbf\" (UID: \"a4957aa1-9cf4-499e-af91-57083b2288d6\") " pod="openshift-insights/insights-runtime-extractor-4xsbf"
Apr 22 19:58:57.316756 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.316738    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh8tp\" (UniqueName: \"kubernetes.io/projected/cf9153e8-d813-4e2c-bd0a-1140521bf7df-kube-api-access-nh8tp\") pod \"downloads-6bcc868b7-fjzng\" (UID: \"cf9153e8-d813-4e2c-bd0a-1140521bf7df\") " pod="openshift-console/downloads-6bcc868b7-fjzng"
Apr 22 19:58:57.329472 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.329450    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4xsbf"
Apr 22 19:58:57.399462 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.399426    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-pp5hw"
Apr 22 19:58:57.400102 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.400080    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f66e796-5123-4e93-bfe8-da6796c18fe9-registry-tls\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.400228 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.400123    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6f66e796-5123-4e93-bfe8-da6796c18fe9-ca-trust-extracted\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.400228 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.400171    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6f66e796-5123-4e93-bfe8-da6796c18fe9-trusted-ca\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.400553 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.400531    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6f66e796-5123-4e93-bfe8-da6796c18fe9-ca-trust-extracted\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.400700 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.400671    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6f66e796-5123-4e93-bfe8-da6796c18fe9-image-registry-private-configuration\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.400876 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.400719    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6f66e796-5123-4e93-bfe8-da6796c18fe9-registry-certificates\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.400876 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.400746    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-255mh\" (UniqueName: \"kubernetes.io/projected/6f66e796-5123-4e93-bfe8-da6796c18fe9-kube-api-access-255mh\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.400876 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.400785    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f66e796-5123-4e93-bfe8-da6796c18fe9-bound-sa-token\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.400876 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.400819    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6f66e796-5123-4e93-bfe8-da6796c18fe9-installation-pull-secrets\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.401585 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.401366    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6f66e796-5123-4e93-bfe8-da6796c18fe9-trusted-ca\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.401585 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.401495    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6f66e796-5123-4e93-bfe8-da6796c18fe9-registry-certificates\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.403093 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.403073    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f66e796-5123-4e93-bfe8-da6796c18fe9-registry-tls\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.405294 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.405271    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6f66e796-5123-4e93-bfe8-da6796c18fe9-image-registry-private-configuration\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.405405 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.405385    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6bcc868b7-fjzng"
Apr 22 19:58:57.405590 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.405511    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6f66e796-5123-4e93-bfe8-da6796c18fe9-installation-pull-secrets\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.418683 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.418656    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f66e796-5123-4e93-bfe8-da6796c18fe9-bound-sa-token\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.419333 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.419309    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-255mh\" (UniqueName: \"kubernetes.io/projected/6f66e796-5123-4e93-bfe8-da6796c18fe9-kube-api-access-255mh\") pod \"image-registry-5db49c5564-f22ph\" (UID: \"6f66e796-5123-4e93-bfe8-da6796c18fe9\") " pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.462241 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.462212    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4xsbf"]
Apr 22 19:58:57.467657 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:58:57.467623    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4957aa1_9cf4_499e_af91_57083b2288d6.slice/crio-9b4d9a362944dd16817687fd069ea6516bc1c6b1da7c4549b0ee2e5c502002a7 WatchSource:0}: Error finding container 9b4d9a362944dd16817687fd069ea6516bc1c6b1da7c4549b0ee2e5c502002a7: Status 404 returned error can't find the container with id 9b4d9a362944dd16817687fd069ea6516bc1c6b1da7c4549b0ee2e5c502002a7
Apr 22 19:58:57.499817 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.499791    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.545521 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.545463    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-pp5hw"]
Apr 22 19:58:57.548799 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:58:57.548773    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f4ac74_fc98_4d51_8bb7_136b4695a7bb.slice/crio-1662494d76a15a46b7719c68d2c2bc17951e1a8786966655628b5142c6a4d365 WatchSource:0}: Error finding container 1662494d76a15a46b7719c68d2c2bc17951e1a8786966655628b5142c6a4d365: Status 404 returned error can't find the container with id 1662494d76a15a46b7719c68d2c2bc17951e1a8786966655628b5142c6a4d365
Apr 22 19:58:57.560918 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.560743    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6bcc868b7-fjzng"]
Apr 22 19:58:57.642367 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.642323    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5db49c5564-f22ph"]
Apr 22 19:58:57.645568 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:58:57.645526    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f66e796_5123_4e93_bfe8_da6796c18fe9.slice/crio-ddcfa3076c3eba63d05d9bea166d4cc0db991ce4c0bd7dd1703f73e0ef94e598 WatchSource:0}: Error finding container ddcfa3076c3eba63d05d9bea166d4cc0db991ce4c0bd7dd1703f73e0ef94e598: Status 404 returned error can't find the container with id ddcfa3076c3eba63d05d9bea166d4cc0db991ce4c0bd7dd1703f73e0ef94e598
Apr 22 19:58:57.726818 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.726787    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4xsbf" event={"ID":"a4957aa1-9cf4-499e-af91-57083b2288d6","Type":"ContainerStarted","Data":"0a1d0368fd3b792deabc60251c042d8dfeaa1f37736cbbbcc73b9901b2ecef71"}
Apr 22 19:58:57.726944 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.726828    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4xsbf" event={"ID":"a4957aa1-9cf4-499e-af91-57083b2288d6","Type":"ContainerStarted","Data":"9b4d9a362944dd16817687fd069ea6516bc1c6b1da7c4549b0ee2e5c502002a7"}
Apr 22 19:58:57.728102 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.728078    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5db49c5564-f22ph" event={"ID":"6f66e796-5123-4e93-bfe8-da6796c18fe9","Type":"ContainerStarted","Data":"8682227671dc79d3307b0efc84eed144bd46ae8b528048d7bd66d275a27365a4"}
Apr 22 19:58:57.728253 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.728108    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5db49c5564-f22ph" event={"ID":"6f66e796-5123-4e93-bfe8-da6796c18fe9","Type":"ContainerStarted","Data":"ddcfa3076c3eba63d05d9bea166d4cc0db991ce4c0bd7dd1703f73e0ef94e598"}
Apr 22 19:58:57.728314 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.728282    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:58:57.729139 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.729107    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6bcc868b7-fjzng" event={"ID":"cf9153e8-d813-4e2c-bd0a-1140521bf7df","Type":"ContainerStarted","Data":"67b55c5e2f3287012539120684bb5f5bb99f9c5e581b119f0d63c26c19fd72c2"}
Apr 22 19:58:57.730130 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.730107    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-pp5hw" event={"ID":"d8f4ac74-fc98-4d51-8bb7-136b4695a7bb","Type":"ContainerStarted","Data":"1662494d76a15a46b7719c68d2c2bc17951e1a8786966655628b5142c6a4d365"}
Apr 22 19:58:57.748336 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:57.748288    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5db49c5564-f22ph" podStartSLOduration=0.748277649 podStartE2EDuration="748.277649ms" podCreationTimestamp="2026-04-22 19:58:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 19:58:57.747825469 +0000 UTC m=+72.847781733" watchObservedRunningTime="2026-04-22 19:58:57.748277649 +0000 UTC m=+72.848233904"
Apr 22 19:58:58.738643 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:58.738608    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4xsbf" event={"ID":"a4957aa1-9cf4-499e-af91-57083b2288d6","Type":"ContainerStarted","Data":"bf22a0e3f4a420f1726acf17b270cf3ed165222ea652daa4d9a1c51669a470f5"}
Apr 22 19:58:59.744062 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:59.744020    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-pp5hw" event={"ID":"d8f4ac74-fc98-4d51-8bb7-136b4695a7bb","Type":"ContainerStarted","Data":"a363c1a645dd3ffc657d354df2aa2c0b0e01472c25b6861843b68d4957a118d5"}
Apr 22 19:58:59.744525 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:59.744389    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-pp5hw"
Apr 22 19:58:59.750031 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:59.750009    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-pp5hw"
Apr 22 19:58:59.759652 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:58:59.759605    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-pp5hw" podStartSLOduration=1.6087348320000001 podStartE2EDuration="2.759587985s" podCreationTimestamp="2026-04-22 19:58:57 +0000 UTC" firstStartedPulling="2026-04-22 19:58:57.551396936 +0000 UTC m=+72.651353185" lastFinishedPulling="2026-04-22 19:58:58.702250092 +0000 UTC m=+73.802206338" observedRunningTime="2026-04-22 19:58:59.758802464 +0000 UTC m=+74.858758743" watchObservedRunningTime="2026-04-22 19:58:59.759587985 +0000 UTC m=+74.859544250"
Apr 22 19:59:00.752357 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:00.752318    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4xsbf" event={"ID":"a4957aa1-9cf4-499e-af91-57083b2288d6","Type":"ContainerStarted","Data":"34d14ba3517e7bd69cdb99cf6db338e6e8bac6bc2cffd7fcabbc8d98b648ad34"}
Apr 22 19:59:00.774587 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:00.774523    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4xsbf" podStartSLOduration=2.544892786 podStartE2EDuration="4.774507139s" podCreationTimestamp="2026-04-22 19:58:56 +0000 UTC" firstStartedPulling="2026-04-22 19:58:57.560719498 +0000 UTC m=+72.660675755" lastFinishedPulling="2026-04-22 19:58:59.790333862 +0000 UTC m=+74.890290108" observedRunningTime="2026-04-22 19:59:00.773385584 +0000 UTC m=+75.873341849" watchObservedRunningTime="2026-04-22 19:59:00.774507139 +0000 UTC m=+75.874463406"
Apr 22 19:59:03.334953 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.334835    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-85d66f5c47-mx8kq"]
Apr 22 19:59:03.339078 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.339051    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:03.342017 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.341994    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Apr 22 19:59:03.342017 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.342012    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Apr 22 19:59:03.342237 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.342042    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8b4lr\""
Apr 22 19:59:03.342237 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.342054    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Apr 22 19:59:03.342237 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.342043    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Apr 22 19:59:03.342237 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.342140    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Apr 22 19:59:03.346314 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.346271    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85d66f5c47-mx8kq"]
Apr 22 19:59:03.445219 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.445189    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1a2af40-9737-49ae-be59-77c15a5c2774-console-config\") pod \"console-85d66f5c47-mx8kq\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") " pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:03.445408 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.445274    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxqbn\" (UniqueName: \"kubernetes.io/projected/c1a2af40-9737-49ae-be59-77c15a5c2774-kube-api-access-pxqbn\") pod \"console-85d66f5c47-mx8kq\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") " pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:03.445408 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.445331    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a2af40-9737-49ae-be59-77c15a5c2774-service-ca\") pod \"console-85d66f5c47-mx8kq\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") " pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:03.445408 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.445367    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1a2af40-9737-49ae-be59-77c15a5c2774-console-oauth-config\") pod \"console-85d66f5c47-mx8kq\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") " pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:03.445408 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.445390    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1a2af40-9737-49ae-be59-77c15a5c2774-oauth-serving-cert\") pod \"console-85d66f5c47-mx8kq\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") " pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:03.445408 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.445412    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1a2af40-9737-49ae-be59-77c15a5c2774-console-serving-cert\") pod \"console-85d66f5c47-mx8kq\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") " pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:03.546730 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.546690    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1a2af40-9737-49ae-be59-77c15a5c2774-console-config\") pod \"console-85d66f5c47-mx8kq\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") " pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:03.546922 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.546748    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pxqbn\" (UniqueName: \"kubernetes.io/projected/c1a2af40-9737-49ae-be59-77c15a5c2774-kube-api-access-pxqbn\") pod \"console-85d66f5c47-mx8kq\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") " pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:03.546922 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.546785    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a2af40-9737-49ae-be59-77c15a5c2774-service-ca\") pod \"console-85d66f5c47-mx8kq\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") " pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:03.546922 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.546821    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1a2af40-9737-49ae-be59-77c15a5c2774-console-oauth-config\") pod \"console-85d66f5c47-mx8kq\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") " pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:03.546922 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.546844    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1a2af40-9737-49ae-be59-77c15a5c2774-oauth-serving-cert\") pod \"console-85d66f5c47-mx8kq\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") " pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:03.546922 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.546869    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1a2af40-9737-49ae-be59-77c15a5c2774-console-serving-cert\") pod \"console-85d66f5c47-mx8kq\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") " pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:03.547581 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.547550    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1a2af40-9737-49ae-be59-77c15a5c2774-console-config\") pod \"console-85d66f5c47-mx8kq\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") " pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:03.547691 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.547635    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1a2af40-9737-49ae-be59-77c15a5c2774-oauth-serving-cert\") pod \"console-85d66f5c47-mx8kq\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") " pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:03.547991 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.547966    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a2af40-9737-49ae-be59-77c15a5c2774-service-ca\") pod \"console-85d66f5c47-mx8kq\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") " pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:03.549625 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.549607    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1a2af40-9737-49ae-be59-77c15a5c2774-console-oauth-config\") pod \"console-85d66f5c47-mx8kq\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") " pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:03.549734 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.549715    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1a2af40-9737-49ae-be59-77c15a5c2774-console-serving-cert\") pod \"console-85d66f5c47-mx8kq\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") " pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:03.556990 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.556963    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxqbn\" (UniqueName: \"kubernetes.io/projected/c1a2af40-9737-49ae-be59-77c15a5c2774-kube-api-access-pxqbn\") pod \"console-85d66f5c47-mx8kq\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") " pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:03.649748 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.649708    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:03.786858 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:03.786814    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85d66f5c47-mx8kq"]
Apr 22 19:59:03.790843 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:59:03.790811    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a2af40_9737_49ae_be59_77c15a5c2774.slice/crio-16db7a61f8aa146839644bc0d26fc6deb69376504e7dbbb5d65af8a3428403da WatchSource:0}: Error finding container 16db7a61f8aa146839644bc0d26fc6deb69376504e7dbbb5d65af8a3428403da: Status 404 returned error can't find the container with id 16db7a61f8aa146839644bc0d26fc6deb69376504e7dbbb5d65af8a3428403da
Apr 22 19:59:04.724335 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:04.724306    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6hc62"
Apr 22 19:59:04.766335 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:04.766299    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85d66f5c47-mx8kq" event={"ID":"c1a2af40-9737-49ae-be59-77c15a5c2774","Type":"ContainerStarted","Data":"16db7a61f8aa146839644bc0d26fc6deb69376504e7dbbb5d65af8a3428403da"}
Apr 22 19:59:05.134307 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.134272    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5"]
Apr 22 19:59:05.138033 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.138012    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5"
Apr 22 19:59:05.148294 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.147823    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Apr 22 19:59:05.148294 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.147843    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Apr 22 19:59:05.148294 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.147859    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-82pdb\""
Apr 22 19:59:05.148294 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.148125    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Apr 22 19:59:05.148676 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.148653    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Apr 22 19:59:05.148801 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.148697    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Apr 22 19:59:05.173191 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.173126    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5"]
Apr 22 19:59:05.215024 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.213823    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-24q4t"]
Apr 22 19:59:05.218674 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.217645    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.222947 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.221003    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Apr 22 19:59:05.222947 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.221273    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Apr 22 19:59:05.222947 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.221395    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-q4d8q\""
Apr 22 19:59:05.222947 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.222225    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Apr 22 19:59:05.260982 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.260796    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9d44df66c-cz8f5\" (UID: \"ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5"
Apr 22 19:59:05.260982 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.260856    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb-metrics-client-ca\") pod \"openshift-state-metrics-9d44df66c-cz8f5\" (UID: \"ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5"
Apr 22 19:59:05.260982 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.260925    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9d44df66c-cz8f5\" (UID: \"ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5"
Apr 22 19:59:05.260982 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.260952    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82vcv\" (UniqueName: \"kubernetes.io/projected/ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb-kube-api-access-82vcv\") pod \"openshift-state-metrics-9d44df66c-cz8f5\" (UID: \"ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5"
Apr 22 19:59:05.363060 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.361909    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3024bfe-c134-4366-9106-da71be312c10-metrics-client-ca\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.363060 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.361956    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d3024bfe-c134-4366-9106-da71be312c10-node-exporter-accelerators-collector-config\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.363060 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.362000    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9d44df66c-cz8f5\" (UID: \"ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5"
Apr 22 19:59:05.363060 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.362030    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3024bfe-c134-4366-9106-da71be312c10-sys\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.363060 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.362058    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3024bfe-c134-4366-9106-da71be312c10-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.363060 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.362121    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d3024bfe-c134-4366-9106-da71be312c10-node-exporter-textfile\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.363060 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.362171    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb-metrics-client-ca\") pod \"openshift-state-metrics-9d44df66c-cz8f5\" (UID: \"ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5"
Apr 22 19:59:05.363060 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.362210    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d3024bfe-c134-4366-9106-da71be312c10-node-exporter-wtmp\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.363060 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.362240    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d3024bfe-c134-4366-9106-da71be312c10-root\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.363060 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:59:05.362287    2576 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Apr 22 19:59:05.363060 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.362296    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9d44df66c-cz8f5\" (UID: \"ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5"
Apr 22 19:59:05.363060 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.362323    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-82vcv\" (UniqueName: \"kubernetes.io/projected/ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb-kube-api-access-82vcv\") pod \"openshift-state-metrics-9d44df66c-cz8f5\" (UID: \"ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5"
Apr 22 19:59:05.363060 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:59:05.362360    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb-openshift-state-metrics-tls podName:ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb nodeName:}" failed. No retries permitted until 2026-04-22 19:59:05.862338566 +0000 UTC m=+80.962294816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb-openshift-state-metrics-tls") pod "openshift-state-metrics-9d44df66c-cz8f5" (UID: "ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb") : secret "openshift-state-metrics-tls" not found
Apr 22 19:59:05.363060 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.362397    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkwtp\" (UniqueName: \"kubernetes.io/projected/d3024bfe-c134-4366-9106-da71be312c10-kube-api-access-pkwtp\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.363060 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.362432    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3024bfe-c134-4366-9106-da71be312c10-node-exporter-tls\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.363948 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.363248    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb-metrics-client-ca\") pod \"openshift-state-metrics-9d44df66c-cz8f5\" (UID: \"ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5"
Apr 22 19:59:05.372260 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.369888    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9d44df66c-cz8f5\" (UID: \"ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5"
Apr 22 19:59:05.376951 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.375681    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82vcv\" (UniqueName: \"kubernetes.io/projected/ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb-kube-api-access-82vcv\") pod \"openshift-state-metrics-9d44df66c-cz8f5\" (UID: \"ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5"
Apr 22 19:59:05.466528 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.466448    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d3024bfe-c134-4366-9106-da71be312c10-node-exporter-wtmp\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.466528 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.466499    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d3024bfe-c134-4366-9106-da71be312c10-root\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.466727 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.466546    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pkwtp\" (UniqueName: \"kubernetes.io/projected/d3024bfe-c134-4366-9106-da71be312c10-kube-api-access-pkwtp\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.466727 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.466573    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3024bfe-c134-4366-9106-da71be312c10-node-exporter-tls\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.466727 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.466602    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3024bfe-c134-4366-9106-da71be312c10-metrics-client-ca\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.466727 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.466627    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d3024bfe-c134-4366-9106-da71be312c10-node-exporter-accelerators-collector-config\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.466727 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.466674    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3024bfe-c134-4366-9106-da71be312c10-sys\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.466727 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.466698    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3024bfe-c134-4366-9106-da71be312c10-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.466727 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.466726    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d3024bfe-c134-4366-9106-da71be312c10-node-exporter-textfile\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.467074 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.467052    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d3024bfe-c134-4366-9106-da71be312c10-node-exporter-textfile\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.467237 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.467218    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d3024bfe-c134-4366-9106-da71be312c10-node-exporter-wtmp\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.467300 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.467265    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d3024bfe-c134-4366-9106-da71be312c10-root\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.467613 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:59:05.467594    2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Apr 22 19:59:05.467683 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:59:05.467657    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3024bfe-c134-4366-9106-da71be312c10-node-exporter-tls podName:d3024bfe-c134-4366-9106-da71be312c10 nodeName:}" failed. No retries permitted until 2026-04-22 19:59:05.967637298 +0000 UTC m=+81.067593548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d3024bfe-c134-4366-9106-da71be312c10-node-exporter-tls") pod "node-exporter-24q4t" (UID: "d3024bfe-c134-4366-9106-da71be312c10") : secret "node-exporter-tls" not found
Apr 22 19:59:05.468684 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.468261    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3024bfe-c134-4366-9106-da71be312c10-metrics-client-ca\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.468684 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.468338    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3024bfe-c134-4366-9106-da71be312c10-sys\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.470312 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.470285    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d3024bfe-c134-4366-9106-da71be312c10-node-exporter-accelerators-collector-config\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.482026 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.481996    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3024bfe-c134-4366-9106-da71be312c10-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.495052 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.495025    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkwtp\" (UniqueName: \"kubernetes.io/projected/d3024bfe-c134-4366-9106-da71be312c10-kube-api-access-pkwtp\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.871425 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.871379    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9d44df66c-cz8f5\" (UID: \"ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5"
Apr 22 19:59:05.874632 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.874601    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9d44df66c-cz8f5\" (UID: \"ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb\") " pod="openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5"
Apr 22 19:59:05.972347 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:05.972302    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3024bfe-c134-4366-9106-da71be312c10-node-exporter-tls\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:05.972530 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:59:05.972418    2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Apr 22 19:59:05.972530 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:59:05.972502    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3024bfe-c134-4366-9106-da71be312c10-node-exporter-tls podName:d3024bfe-c134-4366-9106-da71be312c10 nodeName:}" failed. No retries permitted until 2026-04-22 19:59:06.972482619 +0000 UTC m=+82.072438872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d3024bfe-c134-4366-9106-da71be312c10-node-exporter-tls") pod "node-exporter-24q4t" (UID: "d3024bfe-c134-4366-9106-da71be312c10") : secret "node-exporter-tls" not found
Apr 22 19:59:06.055171 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.054777    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5"
Apr 22 19:59:06.134826 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.134316    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 22 19:59:06.140795 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.140694    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.143321 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.143297    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Apr 22 19:59:06.143510 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.143486    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Apr 22 19:59:06.143674 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.143657    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gkbgg\""
Apr 22 19:59:06.143763 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.143745    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Apr 22 19:59:06.146015 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.143830    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Apr 22 19:59:06.146015 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.143937    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Apr 22 19:59:06.146015 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.144010    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Apr 22 19:59:06.146015 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.144269    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Apr 22 19:59:06.146015 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.144328    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Apr 22 19:59:06.146015 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.144438    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Apr 22 19:59:06.152553 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.152515    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 22 19:59:06.216966 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.216922    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5"]
Apr 22 19:59:06.219769 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:59:06.219730    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7ecfaa_5ca6_4e9a_9a2c_23f8946a38cb.slice/crio-962e2d7f62a6d348a99ce1694118cfe5156977fec27821fd90be4c6da7d5d8aa WatchSource:0}: Error finding container 962e2d7f62a6d348a99ce1694118cfe5156977fec27821fd90be4c6da7d5d8aa: Status 404 returned error can't find the container with id 962e2d7f62a6d348a99ce1694118cfe5156977fec27821fd90be4c6da7d5d8aa
Apr 22 19:59:06.275696 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.275664    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.275780 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.275723    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f046126-596a-4f1c-9d7b-859eef11efb1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.275780 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.275761    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f046126-596a-4f1c-9d7b-859eef11efb1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.275859 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.275797    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.275859 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.275841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.275920 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.275867    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrsgs\" (UniqueName: \"kubernetes.io/projected/5f046126-596a-4f1c-9d7b-859eef11efb1-kube-api-access-hrsgs\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.275920 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.275901    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-web-config\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.275978 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.275929    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.275978 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.275956    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5f046126-596a-4f1c-9d7b-859eef11efb1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.276035 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.275984    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f046126-596a-4f1c-9d7b-859eef11efb1-config-out\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.276035 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.276007    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.276249 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.276034    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f046126-596a-4f1c-9d7b-859eef11efb1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.276249 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.276071    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-config-volume\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.377046 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.376960    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f046126-596a-4f1c-9d7b-859eef11efb1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.377046 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.377021    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.377293 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.377066    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.377293 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.377104    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hrsgs\" (UniqueName: \"kubernetes.io/projected/5f046126-596a-4f1c-9d7b-859eef11efb1-kube-api-access-hrsgs\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.377293 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.377134    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-web-config\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.377293 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.377178    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.377293 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.377202    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5f046126-596a-4f1c-9d7b-859eef11efb1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.377293 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.377246    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f046126-596a-4f1c-9d7b-859eef11efb1-config-out\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.377293 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.377270    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.377862 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.377301    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f046126-596a-4f1c-9d7b-859eef11efb1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.377862 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.377342    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-config-volume\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.377862 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.377391    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.377862 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.377429    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f046126-596a-4f1c-9d7b-859eef11efb1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.378184 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:59:06.378044    2576 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Apr 22 19:59:06.386545 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.386260    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.387231 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.387121    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f046126-596a-4f1c-9d7b-859eef11efb1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.387231 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.387122    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.388210 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.387596    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-web-config\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.388210 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.387812    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.388210 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.388002    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-config-volume\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.388210 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.388062    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f046126-596a-4f1c-9d7b-859eef11efb1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.388210 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:59:06.388098    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-main-tls podName:5f046126-596a-4f1c-9d7b-859eef11efb1 nodeName:}" failed. No retries permitted until 2026-04-22 19:59:06.888073908 +0000 UTC m=+81.988030158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "5f046126-596a-4f1c-9d7b-859eef11efb1") : secret "alertmanager-main-tls" not found
Apr 22 19:59:06.388580 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.388551    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5f046126-596a-4f1c-9d7b-859eef11efb1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.389244 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.389221    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.393693 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.393674    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f046126-596a-4f1c-9d7b-859eef11efb1-config-out\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.394450 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.394387    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrsgs\" (UniqueName: \"kubernetes.io/projected/5f046126-596a-4f1c-9d7b-859eef11efb1-kube-api-access-hrsgs\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.401839 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.401789    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f046126-596a-4f1c-9d7b-859eef11efb1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.775225 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.775114    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5" event={"ID":"ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb","Type":"ContainerStarted","Data":"70b4828290a81dfd95130c435e23890d484129751e2e6b2e6716ed394cd44b39"}
Apr 22 19:59:06.775225 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.775183    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5" event={"ID":"ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb","Type":"ContainerStarted","Data":"d9950d273a6c499205edac5d80253e1349d6539346211182699f1654a597295a"}
Apr 22 19:59:06.775225 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.775197    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5" event={"ID":"ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb","Type":"ContainerStarted","Data":"962e2d7f62a6d348a99ce1694118cfe5156977fec27821fd90be4c6da7d5d8aa"}
Apr 22 19:59:06.981871 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.981830    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:06.982323 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.981899    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3024bfe-c134-4366-9106-da71be312c10-node-exporter-tls\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:06.984796 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.984771    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3024bfe-c134-4366-9106-da71be312c10-node-exporter-tls\") pod \"node-exporter-24q4t\" (UID: \"d3024bfe-c134-4366-9106-da71be312c10\") " pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:06.988053 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:06.988009    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:07.029949 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.029880    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-24q4t"
Apr 22 19:59:07.056505 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.056475    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Apr 22 19:59:07.153382 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.151587    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"]
Apr 22 19:59:07.157570 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.157053    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.159422 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.159396    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Apr 22 19:59:07.159533 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.159439    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Apr 22 19:59:07.159695 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.159669    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5dj7tiuej7oth\""
Apr 22 19:59:07.159695 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.159669    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Apr 22 19:59:07.160121 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.160101    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Apr 22 19:59:07.160609 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.160328    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fdzp6\""
Apr 22 19:59:07.160609 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.160484    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Apr 22 19:59:07.170708 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.170683    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"]
Apr 22 19:59:07.283540 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.283456    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-secret-grpc-tls\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.283540 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.283511    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.283738 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.283605    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.283738 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.283676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-metrics-client-ca\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.283821 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.283752    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.283821 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.283784    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.283894 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.283824    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmtwp\" (UniqueName: \"kubernetes.io/projected/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-kube-api-access-cmtwp\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.283936 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.283911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-secret-thanos-querier-tls\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.385139 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.385106    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-secret-thanos-querier-tls\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.385307 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.385198    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-secret-grpc-tls\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.385307 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.385228    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.385307 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.385267    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.385307 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.385297    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-metrics-client-ca\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.385572 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.385339    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.385572 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.385363    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.385572 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.385398    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cmtwp\" (UniqueName: \"kubernetes.io/projected/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-kube-api-access-cmtwp\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.386305 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.386251    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-metrics-client-ca\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.388434 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.388414    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-secret-thanos-querier-tls\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.388838 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.388815    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.388922 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.388832    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.388922 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.388902    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.389123 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.389102    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-secret-grpc-tls\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.389953 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.389925    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.393460 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.393440    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmtwp\" (UniqueName: \"kubernetes.io/projected/825d8bf0-4414-42c0-bf95-61d4ee25dc9a-kube-api-access-cmtwp\") pod \"thanos-querier-5665d7f4f-w8kfx\" (UID: \"825d8bf0-4414-42c0-bf95-61d4ee25dc9a\") " pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:07.469115 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:07.469074    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:11.131009 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.130970    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-7964998694-p6vhp"]
Apr 22 19:59:11.136996 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.136971    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.144712 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.144687    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Apr 22 19:59:11.145457 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.145431    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7964998694-p6vhp"]
Apr 22 19:59:11.221503 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.221469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-oauth-serving-cert\") pod \"console-7964998694-p6vhp\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") " pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.221679 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.221520    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c870e64-3f28-4a98-8be9-600b7f25d79b-console-oauth-config\") pod \"console-7964998694-p6vhp\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") " pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.221679 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.221563    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c870e64-3f28-4a98-8be9-600b7f25d79b-console-serving-cert\") pod \"console-7964998694-p6vhp\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") " pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.221679 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.221656    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-trusted-ca-bundle\") pod \"console-7964998694-p6vhp\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") " pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.221816 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.221683    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-service-ca\") pod \"console-7964998694-p6vhp\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") " pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.221816 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.221703    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnxnj\" (UniqueName: \"kubernetes.io/projected/1c870e64-3f28-4a98-8be9-600b7f25d79b-kube-api-access-vnxnj\") pod \"console-7964998694-p6vhp\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") " pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.221816 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.221752    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-console-config\") pod \"console-7964998694-p6vhp\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") " pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.322937 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.322900    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c870e64-3f28-4a98-8be9-600b7f25d79b-console-serving-cert\") pod \"console-7964998694-p6vhp\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") " pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.323116 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.322967    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-trusted-ca-bundle\") pod \"console-7964998694-p6vhp\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") " pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.323116 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.322991    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-service-ca\") pod \"console-7964998694-p6vhp\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") " pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.323272 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.323179    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vnxnj\" (UniqueName: \"kubernetes.io/projected/1c870e64-3f28-4a98-8be9-600b7f25d79b-kube-api-access-vnxnj\") pod \"console-7964998694-p6vhp\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") " pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.323329 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.323270    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-console-config\") pod \"console-7964998694-p6vhp\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") " pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.323329 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.323320    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-oauth-serving-cert\") pod \"console-7964998694-p6vhp\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") " pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.323417 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.323361    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c870e64-3f28-4a98-8be9-600b7f25d79b-console-oauth-config\") pod \"console-7964998694-p6vhp\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") " pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.324671 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.323715    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-service-ca\") pod \"console-7964998694-p6vhp\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") " pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.324671 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.324195    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-trusted-ca-bundle\") pod \"console-7964998694-p6vhp\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") " pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.324671 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.324628    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-console-config\") pod \"console-7964998694-p6vhp\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") " pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.324671 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.324660    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-oauth-serving-cert\") pod \"console-7964998694-p6vhp\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") " pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.325882 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.325853    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c870e64-3f28-4a98-8be9-600b7f25d79b-console-serving-cert\") pod \"console-7964998694-p6vhp\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") " pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.325997 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.325927    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c870e64-3f28-4a98-8be9-600b7f25d79b-console-oauth-config\") pod \"console-7964998694-p6vhp\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") " pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.337103 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.337075    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnxnj\" (UniqueName: \"kubernetes.io/projected/1c870e64-3f28-4a98-8be9-600b7f25d79b-kube-api-access-vnxnj\") pod \"console-7964998694-p6vhp\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") " pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:11.448572 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:11.448491    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:14.700381 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:14.700325    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 22 19:59:14.707285 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:59:14.707257    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f046126_596a_4f1c_9d7b_859eef11efb1.slice/crio-9e2d66ea17055b75c4c7eca212c5d7b04cd715067552ed07f73614d04ba3c096 WatchSource:0}: Error finding container 9e2d66ea17055b75c4c7eca212c5d7b04cd715067552ed07f73614d04ba3c096: Status 404 returned error can't find the container with id 9e2d66ea17055b75c4c7eca212c5d7b04cd715067552ed07f73614d04ba3c096
Apr 22 19:59:14.714966 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:14.714937    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"]
Apr 22 19:59:14.718225 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:59:14.718191    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825d8bf0_4414_42c0_bf95_61d4ee25dc9a.slice/crio-7306e001829ac614adc2ae9f676fb76885e9d40d02c16ade6d2a5426da8f78e7 WatchSource:0}: Error finding container 7306e001829ac614adc2ae9f676fb76885e9d40d02c16ade6d2a5426da8f78e7: Status 404 returned error can't find the container with id 7306e001829ac614adc2ae9f676fb76885e9d40d02c16ade6d2a5426da8f78e7
Apr 22 19:59:14.731685 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:14.731658    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7964998694-p6vhp"]
Apr 22 19:59:14.734892 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:59:14.734848    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c870e64_3f28_4a98_8be9_600b7f25d79b.slice/crio-d839548be9f3f3a185fbcfc33e1f619d0f8ed96e6a173d98f51e0c18ff01c899 WatchSource:0}: Error finding container d839548be9f3f3a185fbcfc33e1f619d0f8ed96e6a173d98f51e0c18ff01c899: Status 404 returned error can't find the container with id d839548be9f3f3a185fbcfc33e1f619d0f8ed96e6a173d98f51e0c18ff01c899
Apr 22 19:59:14.800672 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:14.800639    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6bcc868b7-fjzng" event={"ID":"cf9153e8-d813-4e2c-bd0a-1140521bf7df","Type":"ContainerStarted","Data":"d1694b74803e9d66fada029a20d3c644d4587fe5bce4fa825beb7c55cb2c7120"}
Apr 22 19:59:14.800967 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:14.800946    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6bcc868b7-fjzng"
Apr 22 19:59:14.802332 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:14.802012    2576 patch_prober.go:28] interesting pod/downloads-6bcc868b7-fjzng container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.10:8080/\": dial tcp 10.134.0.10:8080: connect: connection refused" start-of-body=
Apr 22 19:59:14.802332 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:14.802072    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6bcc868b7-fjzng" podUID="cf9153e8-d813-4e2c-bd0a-1140521bf7df" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.10:8080/\": dial tcp 10.134.0.10:8080: connect: connection refused"
Apr 22 19:59:14.804207 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:14.804181    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85d66f5c47-mx8kq" event={"ID":"c1a2af40-9737-49ae-be59-77c15a5c2774","Type":"ContainerStarted","Data":"b7c8bde14e2bdeb278ccc62ca55f5a13201a5769e9ead97ed562eb8ae807aa68"}
Apr 22 19:59:14.807026 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:14.806244    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7964998694-p6vhp" event={"ID":"1c870e64-3f28-4a98-8be9-600b7f25d79b","Type":"ContainerStarted","Data":"d839548be9f3f3a185fbcfc33e1f619d0f8ed96e6a173d98f51e0c18ff01c899"}
Apr 22 19:59:14.807838 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:14.807813    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-24q4t" event={"ID":"d3024bfe-c134-4366-9106-da71be312c10","Type":"ContainerStarted","Data":"3e4f399ab6def945603a793e5f30a50c91b9ba6e4778bcbbd9044ca8896be3e9"}
Apr 22 19:59:14.809370 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:14.809299    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx" event={"ID":"825d8bf0-4414-42c0-bf95-61d4ee25dc9a","Type":"ContainerStarted","Data":"7306e001829ac614adc2ae9f676fb76885e9d40d02c16ade6d2a5426da8f78e7"}
Apr 22 19:59:14.810923 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:14.810885    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f046126-596a-4f1c-9d7b-859eef11efb1","Type":"ContainerStarted","Data":"9e2d66ea17055b75c4c7eca212c5d7b04cd715067552ed07f73614d04ba3c096"}
Apr 22 19:59:14.818952 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:14.818898    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-6bcc868b7-fjzng" podStartSLOduration=0.790323091 podStartE2EDuration="17.818882678s" podCreationTimestamp="2026-04-22 19:58:57 +0000 UTC" firstStartedPulling="2026-04-22 19:58:57.575425584 +0000 UTC m=+72.675381829" lastFinishedPulling="2026-04-22 19:59:14.60398516 +0000 UTC m=+89.703941416" observedRunningTime="2026-04-22 19:59:14.817009564 +0000 UTC m=+89.916965829" watchObservedRunningTime="2026-04-22 19:59:14.818882678 +0000 UTC m=+89.918838945"
Apr 22 19:59:14.856038 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:14.855979    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-85d66f5c47-mx8kq" podStartSLOduration=1.114822415 podStartE2EDuration="11.855959026s" podCreationTimestamp="2026-04-22 19:59:03 +0000 UTC" firstStartedPulling="2026-04-22 19:59:03.793194565 +0000 UTC m=+78.893150828" lastFinishedPulling="2026-04-22 19:59:14.534331193 +0000 UTC m=+89.634287439" observedRunningTime="2026-04-22 19:59:14.838482284 +0000 UTC m=+89.938438549" watchObservedRunningTime="2026-04-22 19:59:14.855959026 +0000 UTC m=+89.955915295"
Apr 22 19:59:14.856324 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:14.856291    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-7964998694-p6vhp" podStartSLOduration=3.85628308 podStartE2EDuration="3.85628308s" podCreationTimestamp="2026-04-22 19:59:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 19:59:14.855136218 +0000 UTC m=+89.955092493" watchObservedRunningTime="2026-04-22 19:59:14.85628308 +0000 UTC m=+89.956239343"
Apr 22 19:59:15.816397 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:15.816324    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7964998694-p6vhp" event={"ID":"1c870e64-3f28-4a98-8be9-600b7f25d79b","Type":"ContainerStarted","Data":"8c796a47c5093e764814ffd8d045b5540e6aab47f30cd05bf76ceb8b4e58be8f"}
Apr 22 19:59:15.818299 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:15.818264    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-24q4t" event={"ID":"d3024bfe-c134-4366-9106-da71be312c10","Type":"ContainerStarted","Data":"f976c3e26892ce1cefd123b55e7a5c34fb3dc4dea7f9bd7baa94d3a4436978cf"}
Apr 22 19:59:15.822764 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:15.822737    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5" event={"ID":"ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb","Type":"ContainerStarted","Data":"d77f96946d96580700ab47e308c4858794f12ecec1f52447786e3caab9344594"}
Apr 22 19:59:15.830060 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:15.830039    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6bcc868b7-fjzng"
Apr 22 19:59:15.863485 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:15.863435    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9d44df66c-cz8f5" podStartSLOduration=1.61682244 podStartE2EDuration="10.863416505s" podCreationTimestamp="2026-04-22 19:59:05 +0000 UTC" firstStartedPulling="2026-04-22 19:59:06.363239603 +0000 UTC m=+81.463195850" lastFinishedPulling="2026-04-22 19:59:15.609833669 +0000 UTC m=+90.709789915" observedRunningTime="2026-04-22 19:59:15.86024841 +0000 UTC m=+90.960204674" watchObservedRunningTime="2026-04-22 19:59:15.863416505 +0000 UTC m=+90.963372771"
Apr 22 19:59:16.827784 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:16.827740    2576 generic.go:358] "Generic (PLEG): container finished" podID="d3024bfe-c134-4366-9106-da71be312c10" containerID="f976c3e26892ce1cefd123b55e7a5c34fb3dc4dea7f9bd7baa94d3a4436978cf" exitCode=0
Apr 22 19:59:16.828583 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:16.827831    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-24q4t" event={"ID":"d3024bfe-c134-4366-9106-da71be312c10","Type":"ContainerDied","Data":"f976c3e26892ce1cefd123b55e7a5c34fb3dc4dea7f9bd7baa94d3a4436978cf"}
Apr 22 19:59:17.656653 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.655742    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85d66f5c47-mx8kq"]
Apr 22 19:59:17.680556 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.680532    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-75f55b6d76-phcdc"]
Apr 22 19:59:17.696800 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.696559    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75f55b6d76-phcdc"]
Apr 22 19:59:17.696800 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.696664    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:17.786860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.786764    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-trusted-ca-bundle\") pod \"console-75f55b6d76-phcdc\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") " pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:17.786860 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.786806    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-oauth-serving-cert\") pod \"console-75f55b6d76-phcdc\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") " pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:17.787039 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.786868    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjjmf\" (UniqueName: \"kubernetes.io/projected/afafa982-39e5-4a34-b9c4-6e180a4d7606-kube-api-access-cjjmf\") pod \"console-75f55b6d76-phcdc\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") " pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:17.787039 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.786914    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-service-ca\") pod \"console-75f55b6d76-phcdc\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") " pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:17.787039 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.786967    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/afafa982-39e5-4a34-b9c4-6e180a4d7606-console-serving-cert\") pod \"console-75f55b6d76-phcdc\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") " pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:17.787039 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.786989    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/afafa982-39e5-4a34-b9c4-6e180a4d7606-console-oauth-config\") pod \"console-75f55b6d76-phcdc\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") " pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:17.787239 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.787064    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-console-config\") pod \"console-75f55b6d76-phcdc\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") " pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:17.835799 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.835768    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-24q4t" event={"ID":"d3024bfe-c134-4366-9106-da71be312c10","Type":"ContainerStarted","Data":"fb6e85b2407f38392824ff6abf520b8242df961d79fae61406b5ad34bea2c18a"}
Apr 22 19:59:17.836173 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.835808    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-24q4t" event={"ID":"d3024bfe-c134-4366-9106-da71be312c10","Type":"ContainerStarted","Data":"4911cd99ea56fb73c95acccebd30e3171638805b4ebeee64da10479b85812b83"}
Apr 22 19:59:17.837772 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.837691    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx" event={"ID":"825d8bf0-4414-42c0-bf95-61d4ee25dc9a","Type":"ContainerStarted","Data":"a177b27c23fd4a2b2e08b89dcca8cfc50af672972d3c75956e7aac8aecdda9ae"}
Apr 22 19:59:17.837772 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.837719    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx" event={"ID":"825d8bf0-4414-42c0-bf95-61d4ee25dc9a","Type":"ContainerStarted","Data":"7f9409bae425a2b11c64efed0bab62b0bd8a7d5c55282e0623b17ee0d0e68e3d"}
Apr 22 19:59:17.839902 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.839285    2576 generic.go:358] "Generic (PLEG): container finished" podID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerID="fed2b12daa5610f7dabce5a62f98576eeacaae794eaa13737b9399a77b2e224e" exitCode=0
Apr 22 19:59:17.839902 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.839849    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f046126-596a-4f1c-9d7b-859eef11efb1","Type":"ContainerDied","Data":"fed2b12daa5610f7dabce5a62f98576eeacaae794eaa13737b9399a77b2e224e"}
Apr 22 19:59:17.853502 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.853450    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-24q4t" podStartSLOduration=11.782564731 podStartE2EDuration="12.85343407s" podCreationTimestamp="2026-04-22 19:59:05 +0000 UTC" firstStartedPulling="2026-04-22 19:59:14.542075733 +0000 UTC m=+89.642031976" lastFinishedPulling="2026-04-22 19:59:15.612945064 +0000 UTC m=+90.712901315" observedRunningTime="2026-04-22 19:59:17.852218518 +0000 UTC m=+92.952174784" watchObservedRunningTime="2026-04-22 19:59:17.85343407 +0000 UTC m=+92.953390316"
Apr 22 19:59:17.890022 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.887744    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-trusted-ca-bundle\") pod \"console-75f55b6d76-phcdc\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") " pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:17.890022 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.887782    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-oauth-serving-cert\") pod \"console-75f55b6d76-phcdc\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") " pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:17.890022 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.887805    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cjjmf\" (UniqueName: \"kubernetes.io/projected/afafa982-39e5-4a34-b9c4-6e180a4d7606-kube-api-access-cjjmf\") pod \"console-75f55b6d76-phcdc\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") " pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:17.890022 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.887841    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-service-ca\") pod \"console-75f55b6d76-phcdc\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") " pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:17.890022 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.887950    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/afafa982-39e5-4a34-b9c4-6e180a4d7606-console-serving-cert\") pod \"console-75f55b6d76-phcdc\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") " pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:17.890022 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.888660    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-oauth-serving-cert\") pod \"console-75f55b6d76-phcdc\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") " pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:17.890022 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.888746    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/afafa982-39e5-4a34-b9c4-6e180a4d7606-console-oauth-config\") pod \"console-75f55b6d76-phcdc\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") " pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:17.890022 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.888857    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-trusted-ca-bundle\") pod \"console-75f55b6d76-phcdc\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") " pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:17.890022 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.888909    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-console-config\") pod \"console-75f55b6d76-phcdc\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") " pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:17.892396 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.892378    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/afafa982-39e5-4a34-b9c4-6e180a4d7606-console-serving-cert\") pod \"console-75f55b6d76-phcdc\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") " pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:17.893767 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.893742    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/afafa982-39e5-4a34-b9c4-6e180a4d7606-console-oauth-config\") pod \"console-75f55b6d76-phcdc\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") " pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:17.894041 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.894013    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-service-ca\") pod \"console-75f55b6d76-phcdc\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") " pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:17.894237 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.894212    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-console-config\") pod \"console-75f55b6d76-phcdc\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") " pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:17.898328 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:17.898275    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjjmf\" (UniqueName: \"kubernetes.io/projected/afafa982-39e5-4a34-b9c4-6e180a4d7606-kube-api-access-cjjmf\") pod \"console-75f55b6d76-phcdc\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") " pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:18.048822 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:18.048734    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:18.187177 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:18.187121    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75f55b6d76-phcdc"]
Apr 22 19:59:18.191287 ip-10-0-135-245 kubenswrapper[2576]: W0422 19:59:18.191243    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafafa982_39e5_4a34_b9c4_6e180a4d7606.slice/crio-5ceee12b7daa4d7089e7880daafd72bac7aa0c15026c75437f6bf85294291273 WatchSource:0}: Error finding container 5ceee12b7daa4d7089e7880daafd72bac7aa0c15026c75437f6bf85294291273: Status 404 returned error can't find the container with id 5ceee12b7daa4d7089e7880daafd72bac7aa0c15026c75437f6bf85294291273
Apr 22 19:59:18.745424 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:18.745033    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5db49c5564-f22ph"
Apr 22 19:59:18.847389 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:18.847282    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx" event={"ID":"825d8bf0-4414-42c0-bf95-61d4ee25dc9a","Type":"ContainerStarted","Data":"cd3d701be5b0ee0462cfebaeab4f906a80a6b8635c8994ef08dd0f4a80c7d640"}
Apr 22 19:59:18.851350 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:18.851287    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f55b6d76-phcdc" event={"ID":"afafa982-39e5-4a34-b9c4-6e180a4d7606","Type":"ContainerStarted","Data":"05efca94e336cd6a0aa7990e1edd3bf4f72eecdf377f2d4efffed1b3caa540b6"}
Apr 22 19:59:18.851350 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:18.851321    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f55b6d76-phcdc" event={"ID":"afafa982-39e5-4a34-b9c4-6e180a4d7606","Type":"ContainerStarted","Data":"5ceee12b7daa4d7089e7880daafd72bac7aa0c15026c75437f6bf85294291273"}
Apr 22 19:59:18.873763 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:18.873548    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-75f55b6d76-phcdc" podStartSLOduration=1.8735299520000002 podStartE2EDuration="1.873529952s" podCreationTimestamp="2026-04-22 19:59:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 19:59:18.871103078 +0000 UTC m=+93.971059343" watchObservedRunningTime="2026-04-22 19:59:18.873529952 +0000 UTC m=+93.973486229"
Apr 22 19:59:20.862638 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:20.862552    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx" event={"ID":"825d8bf0-4414-42c0-bf95-61d4ee25dc9a","Type":"ContainerStarted","Data":"0610220252a9d77a57203f70dc10db68f961349e9c7932818bc3dedd3a845653"}
Apr 22 19:59:20.862638 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:20.862596    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx" event={"ID":"825d8bf0-4414-42c0-bf95-61d4ee25dc9a","Type":"ContainerStarted","Data":"bd4e0ef63f771f0e98d8820e9020f82e69f791372a45d1a8c289238faad3a2ba"}
Apr 22 19:59:20.862638 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:20.862608    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx" event={"ID":"825d8bf0-4414-42c0-bf95-61d4ee25dc9a","Type":"ContainerStarted","Data":"6b3ea49de7398316a3965ce783b7c9b324042e69e9e1d54c7ab890dacd89c1e9"}
Apr 22 19:59:20.863250 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:20.862706    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:20.866170 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:20.866122    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f046126-596a-4f1c-9d7b-859eef11efb1","Type":"ContainerStarted","Data":"3e857f9ee60112635dae2ed1bfa58757e624430986f752197f308a3a0d3c1941"}
Apr 22 19:59:20.866170 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:20.866169    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f046126-596a-4f1c-9d7b-859eef11efb1","Type":"ContainerStarted","Data":"7d3b5c310442ef92d791f8133f3a2b8d4f85f3de8a344fd1ba6c3e2e2917d9c9"}
Apr 22 19:59:20.866346 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:20.866185    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f046126-596a-4f1c-9d7b-859eef11efb1","Type":"ContainerStarted","Data":"cd7d55f724dcf33a1dc683b20c063fc27b36c204a5130c944b0481d05576a0a7"}
Apr 22 19:59:20.866346 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:20.866199    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f046126-596a-4f1c-9d7b-859eef11efb1","Type":"ContainerStarted","Data":"58e6a74c8a75c8ce3fd1d22710009a0a2049bfdd21b9735252f5f0f996d61a66"}
Apr 22 19:59:20.866346 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:20.866212    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f046126-596a-4f1c-9d7b-859eef11efb1","Type":"ContainerStarted","Data":"f0e33e743a08fae19c6009e99255a8b68afb837c606c447f9a7fc538a8309640"}
Apr 22 19:59:20.866346 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:20.866223    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f046126-596a-4f1c-9d7b-859eef11efb1","Type":"ContainerStarted","Data":"c2ecb62cdafd02d439ae09f752a27645dd0328267cafa86f020d1e6789f953a7"}
Apr 22 19:59:20.887756 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:20.887700    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx" podStartSLOduration=8.599871827 podStartE2EDuration="13.887684168s" podCreationTimestamp="2026-04-22 19:59:07 +0000 UTC" firstStartedPulling="2026-04-22 19:59:14.720703062 +0000 UTC m=+89.820659308" lastFinishedPulling="2026-04-22 19:59:20.008515397 +0000 UTC m=+95.108471649" observedRunningTime="2026-04-22 19:59:20.886348003 +0000 UTC m=+95.986304270" watchObservedRunningTime="2026-04-22 19:59:20.887684168 +0000 UTC m=+95.987640435"
Apr 22 19:59:20.914684 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:20.914638    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=9.611610648 podStartE2EDuration="14.914625692s" podCreationTimestamp="2026-04-22 19:59:06 +0000 UTC" firstStartedPulling="2026-04-22 19:59:14.70927 +0000 UTC m=+89.809226244" lastFinishedPulling="2026-04-22 19:59:20.012285045 +0000 UTC m=+95.112241288" observedRunningTime="2026-04-22 19:59:20.913374264 +0000 UTC m=+96.013330556" watchObservedRunningTime="2026-04-22 19:59:20.914625692 +0000 UTC m=+96.014581956"
Apr 22 19:59:21.449081 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:21.449044    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:21.449081 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:21.449086    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:21.454463 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:21.454439    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:21.875112 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:21.875079    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:23.650286 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:23.650246    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:25.724622 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:25.724588    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pq8s7"
Apr 22 19:59:26.877254 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:26.877225    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5665d7f4f-w8kfx"
Apr 22 19:59:28.049588 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:28.049553    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:28.049588 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:28.049595    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:28.054651 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:28.054628    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:28.895615 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:28.895589    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 19:59:28.944239 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:28.944210    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7964998694-p6vhp"]
Apr 22 19:59:42.679964 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:42.679890    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-85d66f5c47-mx8kq" podUID="c1a2af40-9737-49ae-be59-77c15a5c2774" containerName="console" containerID="cri-o://b7c8bde14e2bdeb278ccc62ca55f5a13201a5769e9ead97ed562eb8ae807aa68" gracePeriod=15
Apr 22 19:59:42.918595 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:42.918573    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-85d66f5c47-mx8kq_c1a2af40-9737-49ae-be59-77c15a5c2774/console/0.log"
Apr 22 19:59:42.918704 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:42.918646    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:42.931705 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:42.931646    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-85d66f5c47-mx8kq_c1a2af40-9737-49ae-be59-77c15a5c2774/console/0.log"
Apr 22 19:59:42.931705 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:42.931685    2576 generic.go:358] "Generic (PLEG): container finished" podID="c1a2af40-9737-49ae-be59-77c15a5c2774" containerID="b7c8bde14e2bdeb278ccc62ca55f5a13201a5769e9ead97ed562eb8ae807aa68" exitCode=2
Apr 22 19:59:42.931866 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:42.931742    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85d66f5c47-mx8kq" event={"ID":"c1a2af40-9737-49ae-be59-77c15a5c2774","Type":"ContainerDied","Data":"b7c8bde14e2bdeb278ccc62ca55f5a13201a5769e9ead97ed562eb8ae807aa68"}
Apr 22 19:59:42.931866 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:42.931763    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85d66f5c47-mx8kq" event={"ID":"c1a2af40-9737-49ae-be59-77c15a5c2774","Type":"ContainerDied","Data":"16db7a61f8aa146839644bc0d26fc6deb69376504e7dbbb5d65af8a3428403da"}
Apr 22 19:59:42.931866 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:42.931777    2576 scope.go:117] "RemoveContainer" containerID="b7c8bde14e2bdeb278ccc62ca55f5a13201a5769e9ead97ed562eb8ae807aa68"
Apr 22 19:59:42.931866 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:42.931744    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85d66f5c47-mx8kq"
Apr 22 19:59:42.941383 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:42.941274    2576 scope.go:117] "RemoveContainer" containerID="b7c8bde14e2bdeb278ccc62ca55f5a13201a5769e9ead97ed562eb8ae807aa68"
Apr 22 19:59:42.941729 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:59:42.941700    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7c8bde14e2bdeb278ccc62ca55f5a13201a5769e9ead97ed562eb8ae807aa68\": container with ID starting with b7c8bde14e2bdeb278ccc62ca55f5a13201a5769e9ead97ed562eb8ae807aa68 not found: ID does not exist" containerID="b7c8bde14e2bdeb278ccc62ca55f5a13201a5769e9ead97ed562eb8ae807aa68"
Apr 22 19:59:42.941820 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:42.941739    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7c8bde14e2bdeb278ccc62ca55f5a13201a5769e9ead97ed562eb8ae807aa68"} err="failed to get container status \"b7c8bde14e2bdeb278ccc62ca55f5a13201a5769e9ead97ed562eb8ae807aa68\": rpc error: code = NotFound desc = could not find container \"b7c8bde14e2bdeb278ccc62ca55f5a13201a5769e9ead97ed562eb8ae807aa68\": container with ID starting with b7c8bde14e2bdeb278ccc62ca55f5a13201a5769e9ead97ed562eb8ae807aa68 not found: ID does not exist"
Apr 22 19:59:43.018165 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:43.018108    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1a2af40-9737-49ae-be59-77c15a5c2774-console-serving-cert\") pod \"c1a2af40-9737-49ae-be59-77c15a5c2774\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") "
Apr 22 19:59:43.018347 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:43.018193    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1a2af40-9737-49ae-be59-77c15a5c2774-oauth-serving-cert\") pod \"c1a2af40-9737-49ae-be59-77c15a5c2774\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") "
Apr 22 19:59:43.018347 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:43.018229    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxqbn\" (UniqueName: \"kubernetes.io/projected/c1a2af40-9737-49ae-be59-77c15a5c2774-kube-api-access-pxqbn\") pod \"c1a2af40-9737-49ae-be59-77c15a5c2774\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") "
Apr 22 19:59:43.018347 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:43.018251    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1a2af40-9737-49ae-be59-77c15a5c2774-console-oauth-config\") pod \"c1a2af40-9737-49ae-be59-77c15a5c2774\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") "
Apr 22 19:59:43.018347 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:43.018274    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1a2af40-9737-49ae-be59-77c15a5c2774-console-config\") pod \"c1a2af40-9737-49ae-be59-77c15a5c2774\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") "
Apr 22 19:59:43.018347 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:43.018293    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a2af40-9737-49ae-be59-77c15a5c2774-service-ca\") pod \"c1a2af40-9737-49ae-be59-77c15a5c2774\" (UID: \"c1a2af40-9737-49ae-be59-77c15a5c2774\") "
Apr 22 19:59:43.018711 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:43.018679    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a2af40-9737-49ae-be59-77c15a5c2774-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c1a2af40-9737-49ae-be59-77c15a5c2774" (UID: "c1a2af40-9737-49ae-be59-77c15a5c2774"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 19:59:43.018793 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:43.018684    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a2af40-9737-49ae-be59-77c15a5c2774-console-config" (OuterVolumeSpecName: "console-config") pod "c1a2af40-9737-49ae-be59-77c15a5c2774" (UID: "c1a2af40-9737-49ae-be59-77c15a5c2774"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 19:59:43.018793 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:43.018731    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a2af40-9737-49ae-be59-77c15a5c2774-service-ca" (OuterVolumeSpecName: "service-ca") pod "c1a2af40-9737-49ae-be59-77c15a5c2774" (UID: "c1a2af40-9737-49ae-be59-77c15a5c2774"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 19:59:43.020655 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:43.020629    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a2af40-9737-49ae-be59-77c15a5c2774-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c1a2af40-9737-49ae-be59-77c15a5c2774" (UID: "c1a2af40-9737-49ae-be59-77c15a5c2774"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 19:59:43.020757 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:43.020668    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a2af40-9737-49ae-be59-77c15a5c2774-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c1a2af40-9737-49ae-be59-77c15a5c2774" (UID: "c1a2af40-9737-49ae-be59-77c15a5c2774"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 19:59:43.020757 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:43.020718    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a2af40-9737-49ae-be59-77c15a5c2774-kube-api-access-pxqbn" (OuterVolumeSpecName: "kube-api-access-pxqbn") pod "c1a2af40-9737-49ae-be59-77c15a5c2774" (UID: "c1a2af40-9737-49ae-be59-77c15a5c2774"). InnerVolumeSpecName "kube-api-access-pxqbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 19:59:43.119351 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:43.119301    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxqbn\" (UniqueName: \"kubernetes.io/projected/c1a2af40-9737-49ae-be59-77c15a5c2774-kube-api-access-pxqbn\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 19:59:43.119351 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:43.119345    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1a2af40-9737-49ae-be59-77c15a5c2774-console-oauth-config\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 19:59:43.119351 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:43.119356    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1a2af40-9737-49ae-be59-77c15a5c2774-console-config\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 19:59:43.119351 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:43.119365    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a2af40-9737-49ae-be59-77c15a5c2774-service-ca\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 19:59:43.119351 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:43.119373    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1a2af40-9737-49ae-be59-77c15a5c2774-console-serving-cert\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 19:59:43.119637 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:43.119382    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1a2af40-9737-49ae-be59-77c15a5c2774-oauth-serving-cert\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 19:59:43.251979 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:43.251944    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85d66f5c47-mx8kq"]
Apr 22 19:59:43.255948 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:43.255922    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85d66f5c47-mx8kq"]
Apr 22 19:59:43.373355 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:43.373325    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1a2af40-9737-49ae-be59-77c15a5c2774" path="/var/lib/kubelet/pods/c1a2af40-9737-49ae-be59-77c15a5c2774/volumes"
Apr 22 19:59:53.969968 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:53.969909    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-7964998694-p6vhp" podUID="1c870e64-3f28-4a98-8be9-600b7f25d79b" containerName="console" containerID="cri-o://8c796a47c5093e764814ffd8d045b5540e6aab47f30cd05bf76ceb8b4e58be8f" gracePeriod=15
Apr 22 19:59:54.250795 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.250773    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7964998694-p6vhp_1c870e64-3f28-4a98-8be9-600b7f25d79b/console/0.log"
Apr 22 19:59:54.250907 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.250835    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:54.300616 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.300588    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c870e64-3f28-4a98-8be9-600b7f25d79b-console-oauth-config\") pod \"1c870e64-3f28-4a98-8be9-600b7f25d79b\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") "
Apr 22 19:59:54.300751 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.300626    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-service-ca\") pod \"1c870e64-3f28-4a98-8be9-600b7f25d79b\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") "
Apr 22 19:59:54.300751 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.300650    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-oauth-serving-cert\") pod \"1c870e64-3f28-4a98-8be9-600b7f25d79b\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") "
Apr 22 19:59:54.300751 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.300667    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c870e64-3f28-4a98-8be9-600b7f25d79b-console-serving-cert\") pod \"1c870e64-3f28-4a98-8be9-600b7f25d79b\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") "
Apr 22 19:59:54.300930 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.300792    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-console-config\") pod \"1c870e64-3f28-4a98-8be9-600b7f25d79b\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") "
Apr 22 19:59:54.300930 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.300849    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-trusted-ca-bundle\") pod \"1c870e64-3f28-4a98-8be9-600b7f25d79b\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") "
Apr 22 19:59:54.300930 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.300908    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnxnj\" (UniqueName: \"kubernetes.io/projected/1c870e64-3f28-4a98-8be9-600b7f25d79b-kube-api-access-vnxnj\") pod \"1c870e64-3f28-4a98-8be9-600b7f25d79b\" (UID: \"1c870e64-3f28-4a98-8be9-600b7f25d79b\") "
Apr 22 19:59:54.301067 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.301015    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-service-ca" (OuterVolumeSpecName: "service-ca") pod "1c870e64-3f28-4a98-8be9-600b7f25d79b" (UID: "1c870e64-3f28-4a98-8be9-600b7f25d79b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 19:59:54.301141 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.301116    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1c870e64-3f28-4a98-8be9-600b7f25d79b" (UID: "1c870e64-3f28-4a98-8be9-600b7f25d79b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 19:59:54.301264 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.301246    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-service-ca\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 19:59:54.301340 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.301271    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-oauth-serving-cert\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 19:59:54.301407 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.301268    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1c870e64-3f28-4a98-8be9-600b7f25d79b" (UID: "1c870e64-3f28-4a98-8be9-600b7f25d79b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 19:59:54.301485 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.301455    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-console-config" (OuterVolumeSpecName: "console-config") pod "1c870e64-3f28-4a98-8be9-600b7f25d79b" (UID: "1c870e64-3f28-4a98-8be9-600b7f25d79b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 19:59:54.302994 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.302970    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c870e64-3f28-4a98-8be9-600b7f25d79b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1c870e64-3f28-4a98-8be9-600b7f25d79b" (UID: "1c870e64-3f28-4a98-8be9-600b7f25d79b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 19:59:54.303118 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.303100    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c870e64-3f28-4a98-8be9-600b7f25d79b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1c870e64-3f28-4a98-8be9-600b7f25d79b" (UID: "1c870e64-3f28-4a98-8be9-600b7f25d79b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 19:59:54.303118 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.303105    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c870e64-3f28-4a98-8be9-600b7f25d79b-kube-api-access-vnxnj" (OuterVolumeSpecName: "kube-api-access-vnxnj") pod "1c870e64-3f28-4a98-8be9-600b7f25d79b" (UID: "1c870e64-3f28-4a98-8be9-600b7f25d79b"). InnerVolumeSpecName "kube-api-access-vnxnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 19:59:54.401688 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.401640    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-console-config\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 19:59:54.401688 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.401678    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c870e64-3f28-4a98-8be9-600b7f25d79b-trusted-ca-bundle\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 19:59:54.401688 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.401690    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnxnj\" (UniqueName: \"kubernetes.io/projected/1c870e64-3f28-4a98-8be9-600b7f25d79b-kube-api-access-vnxnj\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 19:59:54.401688 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.401699    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c870e64-3f28-4a98-8be9-600b7f25d79b-console-oauth-config\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 19:59:54.401947 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.401709    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c870e64-3f28-4a98-8be9-600b7f25d79b-console-serving-cert\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 19:59:54.971848 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.971823    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7964998694-p6vhp_1c870e64-3f28-4a98-8be9-600b7f25d79b/console/0.log"
Apr 22 19:59:54.972302 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.971864    2576 generic.go:358] "Generic (PLEG): container finished" podID="1c870e64-3f28-4a98-8be9-600b7f25d79b" containerID="8c796a47c5093e764814ffd8d045b5540e6aab47f30cd05bf76ceb8b4e58be8f" exitCode=2
Apr 22 19:59:54.972302 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.971898    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7964998694-p6vhp" event={"ID":"1c870e64-3f28-4a98-8be9-600b7f25d79b","Type":"ContainerDied","Data":"8c796a47c5093e764814ffd8d045b5540e6aab47f30cd05bf76ceb8b4e58be8f"}
Apr 22 19:59:54.972302 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.971927    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7964998694-p6vhp"
Apr 22 19:59:54.972302 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.971941    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7964998694-p6vhp" event={"ID":"1c870e64-3f28-4a98-8be9-600b7f25d79b","Type":"ContainerDied","Data":"d839548be9f3f3a185fbcfc33e1f619d0f8ed96e6a173d98f51e0c18ff01c899"}
Apr 22 19:59:54.972302 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.971956    2576 scope.go:117] "RemoveContainer" containerID="8c796a47c5093e764814ffd8d045b5540e6aab47f30cd05bf76ceb8b4e58be8f"
Apr 22 19:59:54.984386 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.984370    2576 scope.go:117] "RemoveContainer" containerID="8c796a47c5093e764814ffd8d045b5540e6aab47f30cd05bf76ceb8b4e58be8f"
Apr 22 19:59:54.984634 ip-10-0-135-245 kubenswrapper[2576]: E0422 19:59:54.984613    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c796a47c5093e764814ffd8d045b5540e6aab47f30cd05bf76ceb8b4e58be8f\": container with ID starting with 8c796a47c5093e764814ffd8d045b5540e6aab47f30cd05bf76ceb8b4e58be8f not found: ID does not exist" containerID="8c796a47c5093e764814ffd8d045b5540e6aab47f30cd05bf76ceb8b4e58be8f"
Apr 22 19:59:54.984743 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.984639    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c796a47c5093e764814ffd8d045b5540e6aab47f30cd05bf76ceb8b4e58be8f"} err="failed to get container status \"8c796a47c5093e764814ffd8d045b5540e6aab47f30cd05bf76ceb8b4e58be8f\": rpc error: code = NotFound desc = could not find container \"8c796a47c5093e764814ffd8d045b5540e6aab47f30cd05bf76ceb8b4e58be8f\": container with ID starting with 8c796a47c5093e764814ffd8d045b5540e6aab47f30cd05bf76ceb8b4e58be8f not found: ID does not exist"
Apr 22 19:59:54.994030 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.994006    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7964998694-p6vhp"]
Apr 22 19:59:54.997838 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:54.997817    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7964998694-p6vhp"]
Apr 22 19:59:55.373312 ip-10-0-135-245 kubenswrapper[2576]: I0422 19:59:55.373275    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c870e64-3f28-4a98-8be9-600b7f25d79b" path="/var/lib/kubelet/pods/1c870e64-3f28-4a98-8be9-600b7f25d79b/volumes"
Apr 22 20:00:25.579170 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:25.579114    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 22 20:00:25.579590 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:25.579552    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="alertmanager" containerID="cri-o://c2ecb62cdafd02d439ae09f752a27645dd0328267cafa86f020d1e6789f953a7" gracePeriod=120
Apr 22 20:00:25.579653 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:25.579621    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="kube-rbac-proxy-metric" containerID="cri-o://7d3b5c310442ef92d791f8133f3a2b8d4f85f3de8a344fd1ba6c3e2e2917d9c9" gracePeriod=120
Apr 22 20:00:25.579729 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:25.579707    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="config-reloader" containerID="cri-o://f0e33e743a08fae19c6009e99255a8b68afb837c606c447f9a7fc538a8309640" gracePeriod=120
Apr 22 20:00:25.579789 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:25.579745    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="prom-label-proxy" containerID="cri-o://3e857f9ee60112635dae2ed1bfa58757e624430986f752197f308a3a0d3c1941" gracePeriod=120
Apr 22 20:00:25.579789 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:25.579687    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="kube-rbac-proxy-web" containerID="cri-o://58e6a74c8a75c8ce3fd1d22710009a0a2049bfdd21b9735252f5f0f996d61a66" gracePeriod=120
Apr 22 20:00:25.579887 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:25.579764    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="kube-rbac-proxy" containerID="cri-o://cd7d55f724dcf33a1dc683b20c063fc27b36c204a5130c944b0481d05576a0a7" gracePeriod=120
Apr 22 20:00:26.022881 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.022843    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-599b7fb674-24kvs"]
Apr 22 20:00:26.023178 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.023140    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c870e64-3f28-4a98-8be9-600b7f25d79b" containerName="console"
Apr 22 20:00:26.023178 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.023172    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c870e64-3f28-4a98-8be9-600b7f25d79b" containerName="console"
Apr 22 20:00:26.023321 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.023189    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a2af40-9737-49ae-be59-77c15a5c2774" containerName="console"
Apr 22 20:00:26.023321 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.023194    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a2af40-9737-49ae-be59-77c15a5c2774" containerName="console"
Apr 22 20:00:26.023321 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.023241    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="c1a2af40-9737-49ae-be59-77c15a5c2774" containerName="console"
Apr 22 20:00:26.023321 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.023250    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="1c870e64-3f28-4a98-8be9-600b7f25d79b" containerName="console"
Apr 22 20:00:26.025967 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.025950    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.044324 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.044298    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-599b7fb674-24kvs"]
Apr 22 20:00:26.069656 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.069617    2576 generic.go:358] "Generic (PLEG): container finished" podID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerID="3e857f9ee60112635dae2ed1bfa58757e624430986f752197f308a3a0d3c1941" exitCode=0
Apr 22 20:00:26.069656 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.069649    2576 generic.go:358] "Generic (PLEG): container finished" podID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerID="cd7d55f724dcf33a1dc683b20c063fc27b36c204a5130c944b0481d05576a0a7" exitCode=0
Apr 22 20:00:26.069656 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.069656    2576 generic.go:358] "Generic (PLEG): container finished" podID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerID="f0e33e743a08fae19c6009e99255a8b68afb837c606c447f9a7fc538a8309640" exitCode=0
Apr 22 20:00:26.069656 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.069663    2576 generic.go:358] "Generic (PLEG): container finished" podID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerID="c2ecb62cdafd02d439ae09f752a27645dd0328267cafa86f020d1e6789f953a7" exitCode=0
Apr 22 20:00:26.069917 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.069700    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f046126-596a-4f1c-9d7b-859eef11efb1","Type":"ContainerDied","Data":"3e857f9ee60112635dae2ed1bfa58757e624430986f752197f308a3a0d3c1941"}
Apr 22 20:00:26.069917 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.069734    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f046126-596a-4f1c-9d7b-859eef11efb1","Type":"ContainerDied","Data":"cd7d55f724dcf33a1dc683b20c063fc27b36c204a5130c944b0481d05576a0a7"}
Apr 22 20:00:26.069917 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.069743    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f046126-596a-4f1c-9d7b-859eef11efb1","Type":"ContainerDied","Data":"f0e33e743a08fae19c6009e99255a8b68afb837c606c447f9a7fc538a8309640"}
Apr 22 20:00:26.069917 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.069752    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f046126-596a-4f1c-9d7b-859eef11efb1","Type":"ContainerDied","Data":"c2ecb62cdafd02d439ae09f752a27645dd0328267cafa86f020d1e6789f953a7"}
Apr 22 20:00:26.148218 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.148178    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cf3857b3-1362-4623-a6ab-20a6b7a167cd-console-oauth-config\") pod \"console-599b7fb674-24kvs\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") " pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.148218 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.148220    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf3857b3-1362-4623-a6ab-20a6b7a167cd-console-serving-cert\") pod \"console-599b7fb674-24kvs\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") " pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.148432 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.148235    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-console-config\") pod \"console-599b7fb674-24kvs\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") " pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.148432 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.148303    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-service-ca\") pod \"console-599b7fb674-24kvs\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") " pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.148432 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.148343    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-oauth-serving-cert\") pod \"console-599b7fb674-24kvs\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") " pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.148432 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.148366    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-trusted-ca-bundle\") pod \"console-599b7fb674-24kvs\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") " pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.148576 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.148435    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6b2k\" (UniqueName: \"kubernetes.io/projected/cf3857b3-1362-4623-a6ab-20a6b7a167cd-kube-api-access-w6b2k\") pod \"console-599b7fb674-24kvs\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") " pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.249557 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.249524    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-service-ca\") pod \"console-599b7fb674-24kvs\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") " pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.249557 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.249562    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-oauth-serving-cert\") pod \"console-599b7fb674-24kvs\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") " pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.249790 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.249581    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-trusted-ca-bundle\") pod \"console-599b7fb674-24kvs\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") " pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.249790 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.249669    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w6b2k\" (UniqueName: \"kubernetes.io/projected/cf3857b3-1362-4623-a6ab-20a6b7a167cd-kube-api-access-w6b2k\") pod \"console-599b7fb674-24kvs\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") " pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.249790 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.249734    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cf3857b3-1362-4623-a6ab-20a6b7a167cd-console-oauth-config\") pod \"console-599b7fb674-24kvs\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") " pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.249790 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.249768    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf3857b3-1362-4623-a6ab-20a6b7a167cd-console-serving-cert\") pod \"console-599b7fb674-24kvs\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") " pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.249979 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.249794    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-console-config\") pod \"console-599b7fb674-24kvs\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") " pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.250385 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.250350    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-oauth-serving-cert\") pod \"console-599b7fb674-24kvs\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") " pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.250502 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.250358    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-service-ca\") pod \"console-599b7fb674-24kvs\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") " pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.250714 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.250598    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-trusted-ca-bundle\") pod \"console-599b7fb674-24kvs\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") " pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.250763 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.250698    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-console-config\") pod \"console-599b7fb674-24kvs\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") " pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.252324 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.252299    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cf3857b3-1362-4623-a6ab-20a6b7a167cd-console-oauth-config\") pod \"console-599b7fb674-24kvs\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") " pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.252445 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.252426    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf3857b3-1362-4623-a6ab-20a6b7a167cd-console-serving-cert\") pod \"console-599b7fb674-24kvs\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") " pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.257942 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.257920    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6b2k\" (UniqueName: \"kubernetes.io/projected/cf3857b3-1362-4623-a6ab-20a6b7a167cd-kube-api-access-w6b2k\") pod \"console-599b7fb674-24kvs\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") " pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.335000 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.334912    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:26.454018 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.453993    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-599b7fb674-24kvs"]
Apr 22 20:00:26.456554 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:00:26.456530    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3857b3_1362_4623_a6ab_20a6b7a167cd.slice/crio-6d57704d8d122eb67daddcf3cb3c6650ced10e2e1310c4e04ce9eb8199aa4b43 WatchSource:0}: Error finding container 6d57704d8d122eb67daddcf3cb3c6650ced10e2e1310c4e04ce9eb8199aa4b43: Status 404 returned error can't find the container with id 6d57704d8d122eb67daddcf3cb3c6650ced10e2e1310c4e04ce9eb8199aa4b43
Apr 22 20:00:26.824834 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.824811    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:26.955472 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.955373    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrsgs\" (UniqueName: \"kubernetes.io/projected/5f046126-596a-4f1c-9d7b-859eef11efb1-kube-api-access-hrsgs\") pod \"5f046126-596a-4f1c-9d7b-859eef11efb1\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") "
Apr 22 20:00:26.955472 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.955432    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f046126-596a-4f1c-9d7b-859eef11efb1-tls-assets\") pod \"5f046126-596a-4f1c-9d7b-859eef11efb1\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") "
Apr 22 20:00:26.955472 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.955460    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-kube-rbac-proxy\") pod \"5f046126-596a-4f1c-9d7b-859eef11efb1\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") "
Apr 22 20:00:26.955751 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.955634    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-web-config\") pod \"5f046126-596a-4f1c-9d7b-859eef11efb1\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") "
Apr 22 20:00:26.955751 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.955674    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5f046126-596a-4f1c-9d7b-859eef11efb1\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") "
Apr 22 20:00:26.955751 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.955719    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-cluster-tls-config\") pod \"5f046126-596a-4f1c-9d7b-859eef11efb1\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") "
Apr 22 20:00:26.955751 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.955749    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f046126-596a-4f1c-9d7b-859eef11efb1-config-out\") pod \"5f046126-596a-4f1c-9d7b-859eef11efb1\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") "
Apr 22 20:00:26.955962 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.955813    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-kube-rbac-proxy-web\") pod \"5f046126-596a-4f1c-9d7b-859eef11efb1\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") "
Apr 22 20:00:26.955962 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.955842    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f046126-596a-4f1c-9d7b-859eef11efb1-alertmanager-trusted-ca-bundle\") pod \"5f046126-596a-4f1c-9d7b-859eef11efb1\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") "
Apr 22 20:00:26.955962 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.955872    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f046126-596a-4f1c-9d7b-859eef11efb1-metrics-client-ca\") pod \"5f046126-596a-4f1c-9d7b-859eef11efb1\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") "
Apr 22 20:00:26.955962 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.955922    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-config-volume\") pod \"5f046126-596a-4f1c-9d7b-859eef11efb1\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") "
Apr 22 20:00:26.955962 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.955952    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-main-tls\") pod \"5f046126-596a-4f1c-9d7b-859eef11efb1\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") "
Apr 22 20:00:26.956394 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.955981    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5f046126-596a-4f1c-9d7b-859eef11efb1-alertmanager-main-db\") pod \"5f046126-596a-4f1c-9d7b-859eef11efb1\" (UID: \"5f046126-596a-4f1c-9d7b-859eef11efb1\") "
Apr 22 20:00:26.956517 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.956425    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f046126-596a-4f1c-9d7b-859eef11efb1-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "5f046126-596a-4f1c-9d7b-859eef11efb1" (UID: "5f046126-596a-4f1c-9d7b-859eef11efb1"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:00:26.957071 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.956794    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f046126-596a-4f1c-9d7b-859eef11efb1-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "5f046126-596a-4f1c-9d7b-859eef11efb1" (UID: "5f046126-596a-4f1c-9d7b-859eef11efb1"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:00:26.957071 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.956907    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f046126-596a-4f1c-9d7b-859eef11efb1-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "5f046126-596a-4f1c-9d7b-859eef11efb1" (UID: "5f046126-596a-4f1c-9d7b-859eef11efb1"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:00:26.958962 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.958914    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "5f046126-596a-4f1c-9d7b-859eef11efb1" (UID: "5f046126-596a-4f1c-9d7b-859eef11efb1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:00:26.959395 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.959362    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "5f046126-596a-4f1c-9d7b-859eef11efb1" (UID: "5f046126-596a-4f1c-9d7b-859eef11efb1"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:00:26.959395 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.959377    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "5f046126-596a-4f1c-9d7b-859eef11efb1" (UID: "5f046126-596a-4f1c-9d7b-859eef11efb1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:00:26.959552 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.959377    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f046126-596a-4f1c-9d7b-859eef11efb1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5f046126-596a-4f1c-9d7b-859eef11efb1" (UID: "5f046126-596a-4f1c-9d7b-859eef11efb1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:00:26.959754 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.959731    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-config-volume" (OuterVolumeSpecName: "config-volume") pod "5f046126-596a-4f1c-9d7b-859eef11efb1" (UID: "5f046126-596a-4f1c-9d7b-859eef11efb1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:00:26.959854 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.959753    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f046126-596a-4f1c-9d7b-859eef11efb1-kube-api-access-hrsgs" (OuterVolumeSpecName: "kube-api-access-hrsgs") pod "5f046126-596a-4f1c-9d7b-859eef11efb1" (UID: "5f046126-596a-4f1c-9d7b-859eef11efb1"). InnerVolumeSpecName "kube-api-access-hrsgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:00:26.959973 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.959959    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f046126-596a-4f1c-9d7b-859eef11efb1-config-out" (OuterVolumeSpecName: "config-out") pod "5f046126-596a-4f1c-9d7b-859eef11efb1" (UID: "5f046126-596a-4f1c-9d7b-859eef11efb1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:00:26.960185 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.960141    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "5f046126-596a-4f1c-9d7b-859eef11efb1" (UID: "5f046126-596a-4f1c-9d7b-859eef11efb1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:00:26.963423 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.963404    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "5f046126-596a-4f1c-9d7b-859eef11efb1" (UID: "5f046126-596a-4f1c-9d7b-859eef11efb1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:00:26.968848 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:26.968827    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-web-config" (OuterVolumeSpecName: "web-config") pod "5f046126-596a-4f1c-9d7b-859eef11efb1" (UID: "5f046126-596a-4f1c-9d7b-859eef11efb1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:00:27.057703 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.057670    2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-web-config\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:00:27.057703 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.057700    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:00:27.057887 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.057714    2576 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-cluster-tls-config\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:00:27.057887 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.057726    2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f046126-596a-4f1c-9d7b-859eef11efb1-config-out\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:00:27.057887 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.057739    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:00:27.057887 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.057750    2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f046126-596a-4f1c-9d7b-859eef11efb1-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:00:27.057887 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.057764    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f046126-596a-4f1c-9d7b-859eef11efb1-metrics-client-ca\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:00:27.057887 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.057778    2576 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-config-volume\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:00:27.057887 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.057790    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-main-tls\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:00:27.057887 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.057804    2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5f046126-596a-4f1c-9d7b-859eef11efb1-alertmanager-main-db\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:00:27.057887 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.057816    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrsgs\" (UniqueName: \"kubernetes.io/projected/5f046126-596a-4f1c-9d7b-859eef11efb1-kube-api-access-hrsgs\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:00:27.057887 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.057828    2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f046126-596a-4f1c-9d7b-859eef11efb1-tls-assets\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:00:27.057887 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.057842    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f046126-596a-4f1c-9d7b-859eef11efb1-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:00:27.076070 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.076039    2576 generic.go:358] "Generic (PLEG): container finished" podID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerID="7d3b5c310442ef92d791f8133f3a2b8d4f85f3de8a344fd1ba6c3e2e2917d9c9" exitCode=0
Apr 22 20:00:27.076070 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.076062    2576 generic.go:358] "Generic (PLEG): container finished" podID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerID="58e6a74c8a75c8ce3fd1d22710009a0a2049bfdd21b9735252f5f0f996d61a66" exitCode=0
Apr 22 20:00:27.076262 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.076107    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f046126-596a-4f1c-9d7b-859eef11efb1","Type":"ContainerDied","Data":"7d3b5c310442ef92d791f8133f3a2b8d4f85f3de8a344fd1ba6c3e2e2917d9c9"}
Apr 22 20:00:27.076262 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.076139    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f046126-596a-4f1c-9d7b-859eef11efb1","Type":"ContainerDied","Data":"58e6a74c8a75c8ce3fd1d22710009a0a2049bfdd21b9735252f5f0f996d61a66"}
Apr 22 20:00:27.076262 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.076172    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f046126-596a-4f1c-9d7b-859eef11efb1","Type":"ContainerDied","Data":"9e2d66ea17055b75c4c7eca212c5d7b04cd715067552ed07f73614d04ba3c096"}
Apr 22 20:00:27.076262 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.076174    2576 scope.go:117] "RemoveContainer" containerID="3e857f9ee60112635dae2ed1bfa58757e624430986f752197f308a3a0d3c1941"
Apr 22 20:00:27.076262 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.076186    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.077661 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.077631    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599b7fb674-24kvs" event={"ID":"cf3857b3-1362-4623-a6ab-20a6b7a167cd","Type":"ContainerStarted","Data":"eee9d092f1402fa6cd462a39d7cd2f3ef29f9a49ddd7002b6a0c8fff523e2c6c"}
Apr 22 20:00:27.077661 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.077660    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599b7fb674-24kvs" event={"ID":"cf3857b3-1362-4623-a6ab-20a6b7a167cd","Type":"ContainerStarted","Data":"6d57704d8d122eb67daddcf3cb3c6650ced10e2e1310c4e04ce9eb8199aa4b43"}
Apr 22 20:00:27.084212 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.084190    2576 scope.go:117] "RemoveContainer" containerID="7d3b5c310442ef92d791f8133f3a2b8d4f85f3de8a344fd1ba6c3e2e2917d9c9"
Apr 22 20:00:27.091078 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.091062    2576 scope.go:117] "RemoveContainer" containerID="cd7d55f724dcf33a1dc683b20c063fc27b36c204a5130c944b0481d05576a0a7"
Apr 22 20:00:27.097337 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.097320    2576 scope.go:117] "RemoveContainer" containerID="58e6a74c8a75c8ce3fd1d22710009a0a2049bfdd21b9735252f5f0f996d61a66"
Apr 22 20:00:27.103399 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.103381    2576 scope.go:117] "RemoveContainer" containerID="f0e33e743a08fae19c6009e99255a8b68afb837c606c447f9a7fc538a8309640"
Apr 22 20:00:27.109655 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.109638    2576 scope.go:117] "RemoveContainer" containerID="c2ecb62cdafd02d439ae09f752a27645dd0328267cafa86f020d1e6789f953a7"
Apr 22 20:00:27.111955 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.111910    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-599b7fb674-24kvs" podStartSLOduration=1.111892796 podStartE2EDuration="1.111892796s" podCreationTimestamp="2026-04-22 20:00:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:00:27.110016698 +0000 UTC m=+162.209972964" watchObservedRunningTime="2026-04-22 20:00:27.111892796 +0000 UTC m=+162.211849063"
Apr 22 20:00:27.116536 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.116520    2576 scope.go:117] "RemoveContainer" containerID="fed2b12daa5610f7dabce5a62f98576eeacaae794eaa13737b9399a77b2e224e"
Apr 22 20:00:27.122777 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.122761    2576 scope.go:117] "RemoveContainer" containerID="3e857f9ee60112635dae2ed1bfa58757e624430986f752197f308a3a0d3c1941"
Apr 22 20:00:27.123016 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:00:27.123000    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e857f9ee60112635dae2ed1bfa58757e624430986f752197f308a3a0d3c1941\": container with ID starting with 3e857f9ee60112635dae2ed1bfa58757e624430986f752197f308a3a0d3c1941 not found: ID does not exist" containerID="3e857f9ee60112635dae2ed1bfa58757e624430986f752197f308a3a0d3c1941"
Apr 22 20:00:27.123058 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.123026    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e857f9ee60112635dae2ed1bfa58757e624430986f752197f308a3a0d3c1941"} err="failed to get container status \"3e857f9ee60112635dae2ed1bfa58757e624430986f752197f308a3a0d3c1941\": rpc error: code = NotFound desc = could not find container \"3e857f9ee60112635dae2ed1bfa58757e624430986f752197f308a3a0d3c1941\": container with ID starting with 3e857f9ee60112635dae2ed1bfa58757e624430986f752197f308a3a0d3c1941 not found: ID does not exist"
Apr 22 20:00:27.123058 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.123049    2576 scope.go:117] "RemoveContainer" containerID="7d3b5c310442ef92d791f8133f3a2b8d4f85f3de8a344fd1ba6c3e2e2917d9c9"
Apr 22 20:00:27.123402 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:00:27.123382    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d3b5c310442ef92d791f8133f3a2b8d4f85f3de8a344fd1ba6c3e2e2917d9c9\": container with ID starting with 7d3b5c310442ef92d791f8133f3a2b8d4f85f3de8a344fd1ba6c3e2e2917d9c9 not found: ID does not exist" containerID="7d3b5c310442ef92d791f8133f3a2b8d4f85f3de8a344fd1ba6c3e2e2917d9c9"
Apr 22 20:00:27.123482 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.123411    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d3b5c310442ef92d791f8133f3a2b8d4f85f3de8a344fd1ba6c3e2e2917d9c9"} err="failed to get container status \"7d3b5c310442ef92d791f8133f3a2b8d4f85f3de8a344fd1ba6c3e2e2917d9c9\": rpc error: code = NotFound desc = could not find container \"7d3b5c310442ef92d791f8133f3a2b8d4f85f3de8a344fd1ba6c3e2e2917d9c9\": container with ID starting with 7d3b5c310442ef92d791f8133f3a2b8d4f85f3de8a344fd1ba6c3e2e2917d9c9 not found: ID does not exist"
Apr 22 20:00:27.123482 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.123435    2576 scope.go:117] "RemoveContainer" containerID="cd7d55f724dcf33a1dc683b20c063fc27b36c204a5130c944b0481d05576a0a7"
Apr 22 20:00:27.123670 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:00:27.123653    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd7d55f724dcf33a1dc683b20c063fc27b36c204a5130c944b0481d05576a0a7\": container with ID starting with cd7d55f724dcf33a1dc683b20c063fc27b36c204a5130c944b0481d05576a0a7 not found: ID does not exist" containerID="cd7d55f724dcf33a1dc683b20c063fc27b36c204a5130c944b0481d05576a0a7"
Apr 22 20:00:27.123716 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.123677    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd7d55f724dcf33a1dc683b20c063fc27b36c204a5130c944b0481d05576a0a7"} err="failed to get container status \"cd7d55f724dcf33a1dc683b20c063fc27b36c204a5130c944b0481d05576a0a7\": rpc error: code = NotFound desc = could not find container \"cd7d55f724dcf33a1dc683b20c063fc27b36c204a5130c944b0481d05576a0a7\": container with ID starting with cd7d55f724dcf33a1dc683b20c063fc27b36c204a5130c944b0481d05576a0a7 not found: ID does not exist"
Apr 22 20:00:27.123716 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.123693    2576 scope.go:117] "RemoveContainer" containerID="58e6a74c8a75c8ce3fd1d22710009a0a2049bfdd21b9735252f5f0f996d61a66"
Apr 22 20:00:27.123936 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:00:27.123915    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58e6a74c8a75c8ce3fd1d22710009a0a2049bfdd21b9735252f5f0f996d61a66\": container with ID starting with 58e6a74c8a75c8ce3fd1d22710009a0a2049bfdd21b9735252f5f0f996d61a66 not found: ID does not exist" containerID="58e6a74c8a75c8ce3fd1d22710009a0a2049bfdd21b9735252f5f0f996d61a66"
Apr 22 20:00:27.123979 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.123940    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e6a74c8a75c8ce3fd1d22710009a0a2049bfdd21b9735252f5f0f996d61a66"} err="failed to get container status \"58e6a74c8a75c8ce3fd1d22710009a0a2049bfdd21b9735252f5f0f996d61a66\": rpc error: code = NotFound desc = could not find container \"58e6a74c8a75c8ce3fd1d22710009a0a2049bfdd21b9735252f5f0f996d61a66\": container with ID starting with 58e6a74c8a75c8ce3fd1d22710009a0a2049bfdd21b9735252f5f0f996d61a66 not found: ID does not exist"
Apr 22 20:00:27.123979 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.123953    2576 scope.go:117] "RemoveContainer" containerID="f0e33e743a08fae19c6009e99255a8b68afb837c606c447f9a7fc538a8309640"
Apr 22 20:00:27.124215 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:00:27.124197    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e33e743a08fae19c6009e99255a8b68afb837c606c447f9a7fc538a8309640\": container with ID starting with f0e33e743a08fae19c6009e99255a8b68afb837c606c447f9a7fc538a8309640 not found: ID does not exist" containerID="f0e33e743a08fae19c6009e99255a8b68afb837c606c447f9a7fc538a8309640"
Apr 22 20:00:27.124276 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.124222    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e33e743a08fae19c6009e99255a8b68afb837c606c447f9a7fc538a8309640"} err="failed to get container status \"f0e33e743a08fae19c6009e99255a8b68afb837c606c447f9a7fc538a8309640\": rpc error: code = NotFound desc = could not find container \"f0e33e743a08fae19c6009e99255a8b68afb837c606c447f9a7fc538a8309640\": container with ID starting with f0e33e743a08fae19c6009e99255a8b68afb837c606c447f9a7fc538a8309640 not found: ID does not exist"
Apr 22 20:00:27.124276 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.124241    2576 scope.go:117] "RemoveContainer" containerID="c2ecb62cdafd02d439ae09f752a27645dd0328267cafa86f020d1e6789f953a7"
Apr 22 20:00:27.124465 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:00:27.124448    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ecb62cdafd02d439ae09f752a27645dd0328267cafa86f020d1e6789f953a7\": container with ID starting with c2ecb62cdafd02d439ae09f752a27645dd0328267cafa86f020d1e6789f953a7 not found: ID does not exist" containerID="c2ecb62cdafd02d439ae09f752a27645dd0328267cafa86f020d1e6789f953a7"
Apr 22 20:00:27.124507 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.124468    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ecb62cdafd02d439ae09f752a27645dd0328267cafa86f020d1e6789f953a7"} err="failed to get container status \"c2ecb62cdafd02d439ae09f752a27645dd0328267cafa86f020d1e6789f953a7\": rpc error: code = NotFound desc = could not find container \"c2ecb62cdafd02d439ae09f752a27645dd0328267cafa86f020d1e6789f953a7\": container with ID starting with c2ecb62cdafd02d439ae09f752a27645dd0328267cafa86f020d1e6789f953a7 not found: ID does not exist"
Apr 22 20:00:27.124507 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.124481    2576 scope.go:117] "RemoveContainer" containerID="fed2b12daa5610f7dabce5a62f98576eeacaae794eaa13737b9399a77b2e224e"
Apr 22 20:00:27.124709 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:00:27.124695    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fed2b12daa5610f7dabce5a62f98576eeacaae794eaa13737b9399a77b2e224e\": container with ID starting with fed2b12daa5610f7dabce5a62f98576eeacaae794eaa13737b9399a77b2e224e not found: ID does not exist" containerID="fed2b12daa5610f7dabce5a62f98576eeacaae794eaa13737b9399a77b2e224e"
Apr 22 20:00:27.124747 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.124713    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fed2b12daa5610f7dabce5a62f98576eeacaae794eaa13737b9399a77b2e224e"} err="failed to get container status \"fed2b12daa5610f7dabce5a62f98576eeacaae794eaa13737b9399a77b2e224e\": rpc error: code = NotFound desc = could not find container \"fed2b12daa5610f7dabce5a62f98576eeacaae794eaa13737b9399a77b2e224e\": container with ID starting with fed2b12daa5610f7dabce5a62f98576eeacaae794eaa13737b9399a77b2e224e not found: ID does not exist"
Apr 22 20:00:27.124747 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.124725    2576 scope.go:117] "RemoveContainer" containerID="3e857f9ee60112635dae2ed1bfa58757e624430986f752197f308a3a0d3c1941"
Apr 22 20:00:27.124953 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.124931    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e857f9ee60112635dae2ed1bfa58757e624430986f752197f308a3a0d3c1941"} err="failed to get container status \"3e857f9ee60112635dae2ed1bfa58757e624430986f752197f308a3a0d3c1941\": rpc error: code = NotFound desc = could not find container \"3e857f9ee60112635dae2ed1bfa58757e624430986f752197f308a3a0d3c1941\": container with ID starting with 3e857f9ee60112635dae2ed1bfa58757e624430986f752197f308a3a0d3c1941 not found: ID does not exist"
Apr 22 20:00:27.125003 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.124954    2576 scope.go:117] "RemoveContainer" containerID="7d3b5c310442ef92d791f8133f3a2b8d4f85f3de8a344fd1ba6c3e2e2917d9c9"
Apr 22 20:00:27.125135 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.125115    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d3b5c310442ef92d791f8133f3a2b8d4f85f3de8a344fd1ba6c3e2e2917d9c9"} err="failed to get container status \"7d3b5c310442ef92d791f8133f3a2b8d4f85f3de8a344fd1ba6c3e2e2917d9c9\": rpc error: code = NotFound desc = could not find container \"7d3b5c310442ef92d791f8133f3a2b8d4f85f3de8a344fd1ba6c3e2e2917d9c9\": container with ID starting with 7d3b5c310442ef92d791f8133f3a2b8d4f85f3de8a344fd1ba6c3e2e2917d9c9 not found: ID does not exist"
Apr 22 20:00:27.125218 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.125136    2576 scope.go:117] "RemoveContainer" containerID="cd7d55f724dcf33a1dc683b20c063fc27b36c204a5130c944b0481d05576a0a7"
Apr 22 20:00:27.125908 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.125392    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd7d55f724dcf33a1dc683b20c063fc27b36c204a5130c944b0481d05576a0a7"} err="failed to get container status \"cd7d55f724dcf33a1dc683b20c063fc27b36c204a5130c944b0481d05576a0a7\": rpc error: code = NotFound desc = could not find container \"cd7d55f724dcf33a1dc683b20c063fc27b36c204a5130c944b0481d05576a0a7\": container with ID starting with cd7d55f724dcf33a1dc683b20c063fc27b36c204a5130c944b0481d05576a0a7 not found: ID does not exist"
Apr 22 20:00:27.125908 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.125420    2576 scope.go:117] "RemoveContainer" containerID="58e6a74c8a75c8ce3fd1d22710009a0a2049bfdd21b9735252f5f0f996d61a66"
Apr 22 20:00:27.125908 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.125683    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e6a74c8a75c8ce3fd1d22710009a0a2049bfdd21b9735252f5f0f996d61a66"} err="failed to get container status \"58e6a74c8a75c8ce3fd1d22710009a0a2049bfdd21b9735252f5f0f996d61a66\": rpc error: code = NotFound desc = could not find container \"58e6a74c8a75c8ce3fd1d22710009a0a2049bfdd21b9735252f5f0f996d61a66\": container with ID starting with 58e6a74c8a75c8ce3fd1d22710009a0a2049bfdd21b9735252f5f0f996d61a66 not found: ID does not exist"
Apr 22 20:00:27.125908 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.125701    2576 scope.go:117] "RemoveContainer" containerID="f0e33e743a08fae19c6009e99255a8b68afb837c606c447f9a7fc538a8309640"
Apr 22 20:00:27.125908 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.125902    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e33e743a08fae19c6009e99255a8b68afb837c606c447f9a7fc538a8309640"} err="failed to get container status \"f0e33e743a08fae19c6009e99255a8b68afb837c606c447f9a7fc538a8309640\": rpc error: code = NotFound desc = could not find container \"f0e33e743a08fae19c6009e99255a8b68afb837c606c447f9a7fc538a8309640\": container with ID starting with f0e33e743a08fae19c6009e99255a8b68afb837c606c447f9a7fc538a8309640 not found: ID does not exist"
Apr 22 20:00:27.126196 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.125918    2576 scope.go:117] "RemoveContainer" containerID="c2ecb62cdafd02d439ae09f752a27645dd0328267cafa86f020d1e6789f953a7"
Apr 22 20:00:27.126196 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.126123    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ecb62cdafd02d439ae09f752a27645dd0328267cafa86f020d1e6789f953a7"} err="failed to get container status \"c2ecb62cdafd02d439ae09f752a27645dd0328267cafa86f020d1e6789f953a7\": rpc error: code = NotFound desc = could not find container \"c2ecb62cdafd02d439ae09f752a27645dd0328267cafa86f020d1e6789f953a7\": container with ID starting with c2ecb62cdafd02d439ae09f752a27645dd0328267cafa86f020d1e6789f953a7 not found: ID does not exist"
Apr 22 20:00:27.126196 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.126134    2576 scope.go:117] "RemoveContainer" containerID="fed2b12daa5610f7dabce5a62f98576eeacaae794eaa13737b9399a77b2e224e"
Apr 22 20:00:27.126382 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.126366    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fed2b12daa5610f7dabce5a62f98576eeacaae794eaa13737b9399a77b2e224e"} err="failed to get container status \"fed2b12daa5610f7dabce5a62f98576eeacaae794eaa13737b9399a77b2e224e\": rpc error: code = NotFound desc = could not find container \"fed2b12daa5610f7dabce5a62f98576eeacaae794eaa13737b9399a77b2e224e\": container with ID starting with fed2b12daa5610f7dabce5a62f98576eeacaae794eaa13737b9399a77b2e224e not found: ID does not exist"
Apr 22 20:00:27.127473 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.127453    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 22 20:00:27.133754 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.133733    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 22 20:00:27.164917 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.164892    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 22 20:00:27.165192 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.165179    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="kube-rbac-proxy-metric"
Apr 22 20:00:27.165245 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.165194    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="kube-rbac-proxy-metric"
Apr 22 20:00:27.165245 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.165204    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="init-config-reloader"
Apr 22 20:00:27.165245 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.165209    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="init-config-reloader"
Apr 22 20:00:27.165245 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.165217    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="prom-label-proxy"
Apr 22 20:00:27.165245 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.165222    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="prom-label-proxy"
Apr 22 20:00:27.165245 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.165231    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="alertmanager"
Apr 22 20:00:27.165245 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.165236    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="alertmanager"
Apr 22 20:00:27.165245 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.165243    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="config-reloader"
Apr 22 20:00:27.165245 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.165248    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="config-reloader"
Apr 22 20:00:27.165503 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.165255    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="kube-rbac-proxy-web"
Apr 22 20:00:27.165503 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.165260    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="kube-rbac-proxy-web"
Apr 22 20:00:27.165503 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.165267    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="kube-rbac-proxy"
Apr 22 20:00:27.165503 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.165273    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="kube-rbac-proxy"
Apr 22 20:00:27.165503 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.165317    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="config-reloader"
Apr 22 20:00:27.165503 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.165323    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="kube-rbac-proxy"
Apr 22 20:00:27.165503 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.165330    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="alertmanager"
Apr 22 20:00:27.165503 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.165338    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="kube-rbac-proxy-web"
Apr 22 20:00:27.165503 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.165343    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="kube-rbac-proxy-metric"
Apr 22 20:00:27.165503 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.165349    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" containerName="prom-label-proxy"
Apr 22 20:00:27.170195 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.170179    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.173574 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.173545    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Apr 22 20:00:27.173685 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.173546    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Apr 22 20:00:27.173685 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.173621    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Apr 22 20:00:27.173685 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.173678    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Apr 22 20:00:27.173830 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.173713    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Apr 22 20:00:27.173830 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.173716    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Apr 22 20:00:27.173911 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.173849    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Apr 22 20:00:27.174031 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.174018    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gkbgg\""
Apr 22 20:00:27.174680 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.174665    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Apr 22 20:00:27.180943 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.180926    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Apr 22 20:00:27.191837 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.191815    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 22 20:00:27.259530 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.259437    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.259530 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.259479    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-web-config\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.259530 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.259508    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.259750 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.259553    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-config-out\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.259750 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.259582    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-config-volume\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.259750 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.259658    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.259750 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.259703    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb548\" (UniqueName: \"kubernetes.io/projected/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-kube-api-access-bb548\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.259750 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.259726    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-tls-assets\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.259750 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.259747    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.259943 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.259784    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.259943 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.259817    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.259943 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.259838    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.259943 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.259873    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.360788 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.360757    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.360933 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.360800    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.360933 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.360835    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.360933 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.360867    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.361183 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.361129    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.361238 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.361203    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.361296 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.361234    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-web-config\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.361296 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.361273    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.361403 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.361311    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-config-out\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.361403 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.361335    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-config-volume\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.361403 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.361375    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.361552 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.361403    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bb548\" (UniqueName: \"kubernetes.io/projected/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-kube-api-access-bb548\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.361552 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.361429    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-tls-assets\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.361552 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.361452    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.362286 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.361742    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.362346 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.362298    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.364192 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.364166    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.364644 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.364617    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-tls-assets\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.364778 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.364747    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.364880 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.364775    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-config-out\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.364880 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.364850    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-config-volume\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.364880 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.364864    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.365086 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.365069    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.365190 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.365174    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-web-config\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.366056 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.366041    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.372828 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.372800    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb548\" (UniqueName: \"kubernetes.io/projected/92c880f0-b8a6-4dfb-a70c-bae6ec61f048-kube-api-access-bb548\") pod \"alertmanager-main-0\" (UID: \"92c880f0-b8a6-4dfb-a70c-bae6ec61f048\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.376464 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.376436    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f046126-596a-4f1c-9d7b-859eef11efb1" path="/var/lib/kubelet/pods/5f046126-596a-4f1c-9d7b-859eef11efb1/volumes"
Apr 22 20:00:27.480465 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.480423    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Apr 22 20:00:27.630896 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:27.630861    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 22 20:00:27.635719 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:00:27.635692    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c880f0_b8a6_4dfb_a70c_bae6ec61f048.slice/crio-4a2b9d46a2017649d6e09b025267d71e90c5f64650031f777b23b098afc3abce WatchSource:0}: Error finding container 4a2b9d46a2017649d6e09b025267d71e90c5f64650031f777b23b098afc3abce: Status 404 returned error can't find the container with id 4a2b9d46a2017649d6e09b025267d71e90c5f64650031f777b23b098afc3abce
Apr 22 20:00:28.082276 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:28.082236    2576 generic.go:358] "Generic (PLEG): container finished" podID="92c880f0-b8a6-4dfb-a70c-bae6ec61f048" containerID="370a3fc84eec7aa13d3a94a076b34e2fff431e533bff48bb9299e9d1cdbc81c0" exitCode=0
Apr 22 20:00:28.082728 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:28.082335    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92c880f0-b8a6-4dfb-a70c-bae6ec61f048","Type":"ContainerDied","Data":"370a3fc84eec7aa13d3a94a076b34e2fff431e533bff48bb9299e9d1cdbc81c0"}
Apr 22 20:00:28.082728 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:28.082378    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92c880f0-b8a6-4dfb-a70c-bae6ec61f048","Type":"ContainerStarted","Data":"4a2b9d46a2017649d6e09b025267d71e90c5f64650031f777b23b098afc3abce"}
Apr 22 20:00:29.091827 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:29.091794    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92c880f0-b8a6-4dfb-a70c-bae6ec61f048","Type":"ContainerStarted","Data":"c55006e1ee320443c3a8c2351d9d82bb27e148fe37ae821af9edb812fd32e398"}
Apr 22 20:00:29.091827 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:29.091832    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92c880f0-b8a6-4dfb-a70c-bae6ec61f048","Type":"ContainerStarted","Data":"02ec442a64366f55c510a631f3082fc8526493c8310b212a9b9ef0e2d9365705"}
Apr 22 20:00:29.092233 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:29.091847    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92c880f0-b8a6-4dfb-a70c-bae6ec61f048","Type":"ContainerStarted","Data":"5b9342e4817b681f8fb8e6b48b2acb7ef909f9a498e64e0c96055a078444fcf5"}
Apr 22 20:00:29.092233 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:29.091857    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92c880f0-b8a6-4dfb-a70c-bae6ec61f048","Type":"ContainerStarted","Data":"d5dd5fc2db5a2da6ed0c8023f88408497f59eddc3e91f202d788f6d4a96678ff"}
Apr 22 20:00:29.092233 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:29.091865    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92c880f0-b8a6-4dfb-a70c-bae6ec61f048","Type":"ContainerStarted","Data":"2f2f5e619d5a32e41febbabf70442b8d36fb092e428f2e9661813611514d9efa"}
Apr 22 20:00:29.092233 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:29.091874    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92c880f0-b8a6-4dfb-a70c-bae6ec61f048","Type":"ContainerStarted","Data":"355f7bb5815bbba10d22a341e5bcf434592559b30bc2031ea449d9d929505225"}
Apr 22 20:00:29.118901 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:29.118856    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.118841337 podStartE2EDuration="2.118841337s" podCreationTimestamp="2026-04-22 20:00:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:00:29.117956795 +0000 UTC m=+164.217913060" watchObservedRunningTime="2026-04-22 20:00:29.118841337 +0000 UTC m=+164.218797602"
Apr 22 20:00:36.335394 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:36.335295    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:36.335394 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:36.335352    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:36.339895 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:36.339871    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:37.120652 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:37.120625    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:00:37.173696 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:37.173659    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75f55b6d76-phcdc"]
Apr 22 20:00:58.863892 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:58.863853    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-d64wb"]
Apr 22 20:00:58.868406 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:58.868388    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d64wb"
Apr 22 20:00:58.870619 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:58.870595    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Apr 22 20:00:58.874416 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:58.874391    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d64wb"]
Apr 22 20:00:59.022132 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:59.022091    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3d9b1fb8-25a2-4fc7-8083-c6e1ee790339-dbus\") pod \"global-pull-secret-syncer-d64wb\" (UID: \"3d9b1fb8-25a2-4fc7-8083-c6e1ee790339\") " pod="kube-system/global-pull-secret-syncer-d64wb"
Apr 22 20:00:59.022316 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:59.022168    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d9b1fb8-25a2-4fc7-8083-c6e1ee790339-original-pull-secret\") pod \"global-pull-secret-syncer-d64wb\" (UID: \"3d9b1fb8-25a2-4fc7-8083-c6e1ee790339\") " pod="kube-system/global-pull-secret-syncer-d64wb"
Apr 22 20:00:59.022316 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:59.022281    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3d9b1fb8-25a2-4fc7-8083-c6e1ee790339-kubelet-config\") pod \"global-pull-secret-syncer-d64wb\" (UID: \"3d9b1fb8-25a2-4fc7-8083-c6e1ee790339\") " pod="kube-system/global-pull-secret-syncer-d64wb"
Apr 22 20:00:59.123139 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:59.123102    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3d9b1fb8-25a2-4fc7-8083-c6e1ee790339-dbus\") pod \"global-pull-secret-syncer-d64wb\" (UID: \"3d9b1fb8-25a2-4fc7-8083-c6e1ee790339\") " pod="kube-system/global-pull-secret-syncer-d64wb"
Apr 22 20:00:59.123328 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:59.123191    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d9b1fb8-25a2-4fc7-8083-c6e1ee790339-original-pull-secret\") pod \"global-pull-secret-syncer-d64wb\" (UID: \"3d9b1fb8-25a2-4fc7-8083-c6e1ee790339\") " pod="kube-system/global-pull-secret-syncer-d64wb"
Apr 22 20:00:59.123328 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:59.123261    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3d9b1fb8-25a2-4fc7-8083-c6e1ee790339-kubelet-config\") pod \"global-pull-secret-syncer-d64wb\" (UID: \"3d9b1fb8-25a2-4fc7-8083-c6e1ee790339\") " pod="kube-system/global-pull-secret-syncer-d64wb"
Apr 22 20:00:59.123328 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:59.123315    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3d9b1fb8-25a2-4fc7-8083-c6e1ee790339-dbus\") pod \"global-pull-secret-syncer-d64wb\" (UID: \"3d9b1fb8-25a2-4fc7-8083-c6e1ee790339\") " pod="kube-system/global-pull-secret-syncer-d64wb"
Apr 22 20:00:59.123436 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:59.123336    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3d9b1fb8-25a2-4fc7-8083-c6e1ee790339-kubelet-config\") pod \"global-pull-secret-syncer-d64wb\" (UID: \"3d9b1fb8-25a2-4fc7-8083-c6e1ee790339\") " pod="kube-system/global-pull-secret-syncer-d64wb"
Apr 22 20:00:59.125617 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:59.125598    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d9b1fb8-25a2-4fc7-8083-c6e1ee790339-original-pull-secret\") pod \"global-pull-secret-syncer-d64wb\" (UID: \"3d9b1fb8-25a2-4fc7-8083-c6e1ee790339\") " pod="kube-system/global-pull-secret-syncer-d64wb"
Apr 22 20:00:59.178231 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:59.178201    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d64wb"
Apr 22 20:00:59.295751 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:00:59.295717    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d64wb"]
Apr 22 20:00:59.299117 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:00:59.299089    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9b1fb8_25a2_4fc7_8083_c6e1ee790339.slice/crio-54c9b5d5595eeeeaea70d32312a2998f8319b8f9a882260d088b2e843e08452f WatchSource:0}: Error finding container 54c9b5d5595eeeeaea70d32312a2998f8319b8f9a882260d088b2e843e08452f: Status 404 returned error can't find the container with id 54c9b5d5595eeeeaea70d32312a2998f8319b8f9a882260d088b2e843e08452f
Apr 22 20:01:00.180123 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:00.180080    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d64wb" event={"ID":"3d9b1fb8-25a2-4fc7-8083-c6e1ee790339","Type":"ContainerStarted","Data":"54c9b5d5595eeeeaea70d32312a2998f8319b8f9a882260d088b2e843e08452f"}
Apr 22 20:01:02.193523 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:02.193471    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-75f55b6d76-phcdc" podUID="afafa982-39e5-4a34-b9c4-6e180a4d7606" containerName="console" containerID="cri-o://05efca94e336cd6a0aa7990e1edd3bf4f72eecdf377f2d4efffed1b3caa540b6" gracePeriod=15
Apr 22 20:01:02.870465 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:02.870444    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-75f55b6d76-phcdc_afafa982-39e5-4a34-b9c4-6e180a4d7606/console/0.log"
Apr 22 20:01:02.870595 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:02.870503    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 20:01:03.057351 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.057264    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-oauth-serving-cert\") pod \"afafa982-39e5-4a34-b9c4-6e180a4d7606\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") "
Apr 22 20:01:03.057351 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.057327    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-console-config\") pod \"afafa982-39e5-4a34-b9c4-6e180a4d7606\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") "
Apr 22 20:01:03.057581 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.057359    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-trusted-ca-bundle\") pod \"afafa982-39e5-4a34-b9c4-6e180a4d7606\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") "
Apr 22 20:01:03.057581 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.057387    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/afafa982-39e5-4a34-b9c4-6e180a4d7606-console-serving-cert\") pod \"afafa982-39e5-4a34-b9c4-6e180a4d7606\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") "
Apr 22 20:01:03.057581 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.057417    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjjmf\" (UniqueName: \"kubernetes.io/projected/afafa982-39e5-4a34-b9c4-6e180a4d7606-kube-api-access-cjjmf\") pod \"afafa982-39e5-4a34-b9c4-6e180a4d7606\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") "
Apr 22 20:01:03.057581 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.057446    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/afafa982-39e5-4a34-b9c4-6e180a4d7606-console-oauth-config\") pod \"afafa982-39e5-4a34-b9c4-6e180a4d7606\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") "
Apr 22 20:01:03.057581 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.057499    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-service-ca\") pod \"afafa982-39e5-4a34-b9c4-6e180a4d7606\" (UID: \"afafa982-39e5-4a34-b9c4-6e180a4d7606\") "
Apr 22 20:01:03.057796 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.057681    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "afafa982-39e5-4a34-b9c4-6e180a4d7606" (UID: "afafa982-39e5-4a34-b9c4-6e180a4d7606"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:01:03.057796 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.057750    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-console-config" (OuterVolumeSpecName: "console-config") pod "afafa982-39e5-4a34-b9c4-6e180a4d7606" (UID: "afafa982-39e5-4a34-b9c4-6e180a4d7606"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:01:03.057906 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.057787    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "afafa982-39e5-4a34-b9c4-6e180a4d7606" (UID: "afafa982-39e5-4a34-b9c4-6e180a4d7606"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:01:03.057981 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.057960    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-service-ca" (OuterVolumeSpecName: "service-ca") pod "afafa982-39e5-4a34-b9c4-6e180a4d7606" (UID: "afafa982-39e5-4a34-b9c4-6e180a4d7606"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:01:03.059796 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.059762    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afafa982-39e5-4a34-b9c4-6e180a4d7606-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "afafa982-39e5-4a34-b9c4-6e180a4d7606" (UID: "afafa982-39e5-4a34-b9c4-6e180a4d7606"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:01:03.059908 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.059855    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afafa982-39e5-4a34-b9c4-6e180a4d7606-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "afafa982-39e5-4a34-b9c4-6e180a4d7606" (UID: "afafa982-39e5-4a34-b9c4-6e180a4d7606"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:01:03.059908 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.059869    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afafa982-39e5-4a34-b9c4-6e180a4d7606-kube-api-access-cjjmf" (OuterVolumeSpecName: "kube-api-access-cjjmf") pod "afafa982-39e5-4a34-b9c4-6e180a4d7606" (UID: "afafa982-39e5-4a34-b9c4-6e180a4d7606"). InnerVolumeSpecName "kube-api-access-cjjmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:01:03.158459 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.158424    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-trusted-ca-bundle\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:01:03.158459 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.158453    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/afafa982-39e5-4a34-b9c4-6e180a4d7606-console-serving-cert\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:01:03.158459 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.158465    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjjmf\" (UniqueName: \"kubernetes.io/projected/afafa982-39e5-4a34-b9c4-6e180a4d7606-kube-api-access-cjjmf\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:01:03.158669 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.158481    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/afafa982-39e5-4a34-b9c4-6e180a4d7606-console-oauth-config\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:01:03.158669 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.158495    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-service-ca\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:01:03.158669 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.158503    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-oauth-serving-cert\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:01:03.158669 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.158512    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/afafa982-39e5-4a34-b9c4-6e180a4d7606-console-config\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:01:03.191112 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.191077    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d64wb" event={"ID":"3d9b1fb8-25a2-4fc7-8083-c6e1ee790339","Type":"ContainerStarted","Data":"7d02309ed30c7f4c8b0503839cf0a1082296c340fa5c851fc9221bffb1832ced"}
Apr 22 20:01:03.192293 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.192276    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-75f55b6d76-phcdc_afafa982-39e5-4a34-b9c4-6e180a4d7606/console/0.log"
Apr 22 20:01:03.192387 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.192311    2576 generic.go:358] "Generic (PLEG): container finished" podID="afafa982-39e5-4a34-b9c4-6e180a4d7606" containerID="05efca94e336cd6a0aa7990e1edd3bf4f72eecdf377f2d4efffed1b3caa540b6" exitCode=2
Apr 22 20:01:03.192387 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.192361    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f55b6d76-phcdc"
Apr 22 20:01:03.192471 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.192390    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f55b6d76-phcdc" event={"ID":"afafa982-39e5-4a34-b9c4-6e180a4d7606","Type":"ContainerDied","Data":"05efca94e336cd6a0aa7990e1edd3bf4f72eecdf377f2d4efffed1b3caa540b6"}
Apr 22 20:01:03.192471 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.192413    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f55b6d76-phcdc" event={"ID":"afafa982-39e5-4a34-b9c4-6e180a4d7606","Type":"ContainerDied","Data":"5ceee12b7daa4d7089e7880daafd72bac7aa0c15026c75437f6bf85294291273"}
Apr 22 20:01:03.192471 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.192427    2576 scope.go:117] "RemoveContainer" containerID="05efca94e336cd6a0aa7990e1edd3bf4f72eecdf377f2d4efffed1b3caa540b6"
Apr 22 20:01:03.200843 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.200718    2576 scope.go:117] "RemoveContainer" containerID="05efca94e336cd6a0aa7990e1edd3bf4f72eecdf377f2d4efffed1b3caa540b6"
Apr 22 20:01:03.201068 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:03.200986    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05efca94e336cd6a0aa7990e1edd3bf4f72eecdf377f2d4efffed1b3caa540b6\": container with ID starting with 05efca94e336cd6a0aa7990e1edd3bf4f72eecdf377f2d4efffed1b3caa540b6 not found: ID does not exist" containerID="05efca94e336cd6a0aa7990e1edd3bf4f72eecdf377f2d4efffed1b3caa540b6"
Apr 22 20:01:03.201068 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.201012    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05efca94e336cd6a0aa7990e1edd3bf4f72eecdf377f2d4efffed1b3caa540b6"} err="failed to get container status \"05efca94e336cd6a0aa7990e1edd3bf4f72eecdf377f2d4efffed1b3caa540b6\": rpc error: code = NotFound desc = could not find container \"05efca94e336cd6a0aa7990e1edd3bf4f72eecdf377f2d4efffed1b3caa540b6\": container with ID starting with 05efca94e336cd6a0aa7990e1edd3bf4f72eecdf377f2d4efffed1b3caa540b6 not found: ID does not exist"
Apr 22 20:01:03.207040 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.207001    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-d64wb" podStartSLOduration=1.789908592 podStartE2EDuration="5.206989108s" podCreationTimestamp="2026-04-22 20:00:58 +0000 UTC" firstStartedPulling="2026-04-22 20:00:59.300664224 +0000 UTC m=+194.400620467" lastFinishedPulling="2026-04-22 20:01:02.717744736 +0000 UTC m=+197.817700983" observedRunningTime="2026-04-22 20:01:03.205422988 +0000 UTC m=+198.305379252" watchObservedRunningTime="2026-04-22 20:01:03.206989108 +0000 UTC m=+198.306945372"
Apr 22 20:01:03.218669 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.218646    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75f55b6d76-phcdc"]
Apr 22 20:01:03.222215 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.222193    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75f55b6d76-phcdc"]
Apr 22 20:01:03.373094 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:03.373052    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afafa982-39e5-4a34-b9c4-6e180a4d7606" path="/var/lib/kubelet/pods/afafa982-39e5-4a34-b9c4-6e180a4d7606/volumes"
Apr 22 20:01:20.207187 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:20.207133    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm"]
Apr 22 20:01:20.207544 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:20.207440    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afafa982-39e5-4a34-b9c4-6e180a4d7606" containerName="console"
Apr 22 20:01:20.207544 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:20.207454    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="afafa982-39e5-4a34-b9c4-6e180a4d7606" containerName="console"
Apr 22 20:01:20.207544 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:20.207524    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="afafa982-39e5-4a34-b9c4-6e180a4d7606" containerName="console"
Apr 22 20:01:20.211800 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:20.211784    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm"
Apr 22 20:01:20.214294 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:20.214271    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Apr 22 20:01:20.214418 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:20.214399    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4sls\""
Apr 22 20:01:20.215012 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:20.214999    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Apr 22 20:01:20.218737 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:20.218711    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm"]
Apr 22 20:01:20.284448 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:20.284418    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm\" (UID: \"b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm"
Apr 22 20:01:20.284448 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:20.284453    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm\" (UID: \"b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm"
Apr 22 20:01:20.284634 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:20.284519    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69ldl\" (UniqueName: \"kubernetes.io/projected/b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a-kube-api-access-69ldl\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm\" (UID: \"b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm"
Apr 22 20:01:20.384899 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:20.384872    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm\" (UID: \"b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm"
Apr 22 20:01:20.385039 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:20.384906    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm\" (UID: \"b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm"
Apr 22 20:01:20.385039 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:20.384941    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-69ldl\" (UniqueName: \"kubernetes.io/projected/b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a-kube-api-access-69ldl\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm\" (UID: \"b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm"
Apr 22 20:01:20.385302 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:20.385283    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm\" (UID: \"b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm"
Apr 22 20:01:20.385349 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:20.385337    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm\" (UID: \"b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm"
Apr 22 20:01:20.393245 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:20.393223    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69ldl\" (UniqueName: \"kubernetes.io/projected/b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a-kube-api-access-69ldl\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm\" (UID: \"b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm"
Apr 22 20:01:20.521125 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:20.521016    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm"
Apr 22 20:01:20.640844 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:20.640744    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm"]
Apr 22 20:01:20.643559 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:01:20.643522    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f2877d_cdee_4e3e_b0fd_7c405e0fff6a.slice/crio-da533daffb46690b18f60acf0abd9255d67f984118bc8931bb3b4b1fefcf18ce WatchSource:0}: Error finding container da533daffb46690b18f60acf0abd9255d67f984118bc8931bb3b4b1fefcf18ce: Status 404 returned error can't find the container with id da533daffb46690b18f60acf0abd9255d67f984118bc8931bb3b4b1fefcf18ce
Apr 22 20:01:21.245874 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:21.245835    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm" event={"ID":"b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a","Type":"ContainerStarted","Data":"da533daffb46690b18f60acf0abd9255d67f984118bc8931bb3b4b1fefcf18ce"}
Apr 22 20:01:28.271577 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:28.271489    2576 generic.go:358] "Generic (PLEG): container finished" podID="b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a" containerID="346d1c14b94b684546af90386d5015ea2f1f710afb68c22f6b25980d44ef2ff1" exitCode=0
Apr 22 20:01:28.271982 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:28.271579    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm" event={"ID":"b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a","Type":"ContainerDied","Data":"346d1c14b94b684546af90386d5015ea2f1f710afb68c22f6b25980d44ef2ff1"}
Apr 22 20:01:31.283948 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:31.283910    2576 generic.go:358] "Generic (PLEG): container finished" podID="b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a" containerID="f202dabccc59d7634bb36e89e0c29e5e4d78ea94f2755f72a68fb8ae4be827fb" exitCode=0
Apr 22 20:01:31.284456 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:31.284002    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm" event={"ID":"b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a","Type":"ContainerDied","Data":"f202dabccc59d7634bb36e89e0c29e5e4d78ea94f2755f72a68fb8ae4be827fb"}
Apr 22 20:01:39.316738 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:39.316647    2576 generic.go:358] "Generic (PLEG): container finished" podID="b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a" containerID="b096a8a24c8c283bfe91487bc88bf1e2aa1463f290c8755226a3eadd581c62d3" exitCode=0
Apr 22 20:01:39.317135 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:39.316743    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm" event={"ID":"b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a","Type":"ContainerDied","Data":"b096a8a24c8c283bfe91487bc88bf1e2aa1463f290c8755226a3eadd581c62d3"}
Apr 22 20:01:40.442042 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:40.442015    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm"
Apr 22 20:01:40.556991 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:40.556953    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69ldl\" (UniqueName: \"kubernetes.io/projected/b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a-kube-api-access-69ldl\") pod \"b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a\" (UID: \"b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a\") "
Apr 22 20:01:40.557202 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:40.557016    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a-util\") pod \"b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a\" (UID: \"b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a\") "
Apr 22 20:01:40.557202 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:40.557058    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a-bundle\") pod \"b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a\" (UID: \"b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a\") "
Apr 22 20:01:40.557651 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:40.557618    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a-bundle" (OuterVolumeSpecName: "bundle") pod "b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a" (UID: "b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:01:40.559415 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:40.559394    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a-kube-api-access-69ldl" (OuterVolumeSpecName: "kube-api-access-69ldl") pod "b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a" (UID: "b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a"). InnerVolumeSpecName "kube-api-access-69ldl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:01:40.560810 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:40.560784    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a-util" (OuterVolumeSpecName: "util") pod "b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a" (UID: "b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:01:40.658478 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:40.658447    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a-bundle\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:01:40.658478 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:40.658474    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69ldl\" (UniqueName: \"kubernetes.io/projected/b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a-kube-api-access-69ldl\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:01:40.658478 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:40.658484    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a-util\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:01:41.323653 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:41.323625    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm"
Apr 22 20:01:41.323816 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:41.323623    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29ccb9bm" event={"ID":"b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a","Type":"ContainerDied","Data":"da533daffb46690b18f60acf0abd9255d67f984118bc8931bb3b4b1fefcf18ce"}
Apr 22 20:01:41.323816 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:41.323726    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da533daffb46690b18f60acf0abd9255d67f984118bc8931bb3b4b1fefcf18ce"
Apr 22 20:01:47.089050 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.089016    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p"]
Apr 22 20:01:47.089492 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.089477    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a" containerName="pull"
Apr 22 20:01:47.089563 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.089497    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a" containerName="pull"
Apr 22 20:01:47.089563 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.089526    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a" containerName="util"
Apr 22 20:01:47.089563 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.089533    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a" containerName="util"
Apr 22 20:01:47.089563 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.089554    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a" containerName="extract"
Apr 22 20:01:47.089563 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.089562    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a" containerName="extract"
Apr 22 20:01:47.089706 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.089631    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="b4f2877d-cdee-4e3e-b0fd-7c405e0fff6a" containerName="extract"
Apr 22 20:01:47.094031 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.094013    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p"
Apr 22 20:01:47.096807 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.096787    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Apr 22 20:01:47.096930 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.096885    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Apr 22 20:01:47.097518 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.097485    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Apr 22 20:01:47.097631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.097524    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-9g5c7\""
Apr 22 20:01:47.114860 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.114840    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p"]
Apr 22 20:01:47.212516 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.212477    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzp6x\" (UniqueName: \"kubernetes.io/projected/5b605f46-fd42-4fd1-ab7d-f0937667e31c-kube-api-access-wzp6x\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p\" (UID: \"5b605f46-fd42-4fd1-ab7d-f0937667e31c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p"
Apr 22 20:01:47.212516 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.212518    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5b605f46-fd42-4fd1-ab7d-f0937667e31c-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p\" (UID: \"5b605f46-fd42-4fd1-ab7d-f0937667e31c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p"
Apr 22 20:01:47.313046 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.313014    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wzp6x\" (UniqueName: \"kubernetes.io/projected/5b605f46-fd42-4fd1-ab7d-f0937667e31c-kube-api-access-wzp6x\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p\" (UID: \"5b605f46-fd42-4fd1-ab7d-f0937667e31c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p"
Apr 22 20:01:47.313046 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.313048    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5b605f46-fd42-4fd1-ab7d-f0937667e31c-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p\" (UID: \"5b605f46-fd42-4fd1-ab7d-f0937667e31c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p"
Apr 22 20:01:47.315632 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.315602    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5b605f46-fd42-4fd1-ab7d-f0937667e31c-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p\" (UID: \"5b605f46-fd42-4fd1-ab7d-f0937667e31c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p"
Apr 22 20:01:47.322964 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.322937    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzp6x\" (UniqueName: \"kubernetes.io/projected/5b605f46-fd42-4fd1-ab7d-f0937667e31c-kube-api-access-wzp6x\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p\" (UID: \"5b605f46-fd42-4fd1-ab7d-f0937667e31c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p"
Apr 22 20:01:47.403393 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.403365    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p"
Apr 22 20:01:47.540387 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:47.540273    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p"]
Apr 22 20:01:47.543054 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:01:47.543020    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b605f46_fd42_4fd1_ab7d_f0937667e31c.slice/crio-b5c1a61d0fd0847521db3d2c26294af850d86b7329ad1a2681d1a850273fb99c WatchSource:0}: Error finding container b5c1a61d0fd0847521db3d2c26294af850d86b7329ad1a2681d1a850273fb99c: Status 404 returned error can't find the container with id b5c1a61d0fd0847521db3d2c26294af850d86b7329ad1a2681d1a850273fb99c
Apr 22 20:01:48.344381 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:48.344341    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p" event={"ID":"5b605f46-fd42-4fd1-ab7d-f0937667e31c","Type":"ContainerStarted","Data":"b5c1a61d0fd0847521db3d2c26294af850d86b7329ad1a2681d1a850273fb99c"}
Apr 22 20:01:51.354809 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:51.354769    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p" event={"ID":"5b605f46-fd42-4fd1-ab7d-f0937667e31c","Type":"ContainerStarted","Data":"cb98f6699de4b880da2d0b576d11a710fe119746077dd143e7d33cfcc54b10bd"}
Apr 22 20:01:51.355266 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:51.354910    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p"
Apr 22 20:01:51.377793 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:51.377745    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p" podStartSLOduration=0.761492879 podStartE2EDuration="4.377731416s" podCreationTimestamp="2026-04-22 20:01:47 +0000 UTC" firstStartedPulling="2026-04-22 20:01:47.544935969 +0000 UTC m=+242.644892226" lastFinishedPulling="2026-04-22 20:01:51.161174508 +0000 UTC m=+246.261130763" observedRunningTime="2026-04-22 20:01:51.376508246 +0000 UTC m=+246.476464511" watchObservedRunningTime="2026-04-22 20:01:51.377731416 +0000 UTC m=+246.477687681"
Apr 22 20:01:51.665389 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:51.665359    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-f84sg"]
Apr 22 20:01:51.668551 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:51.668531    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-f84sg"
Apr 22 20:01:51.671758 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:51.671740    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-jl2mh\""
Apr 22 20:01:51.672038 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:51.672024    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Apr 22 20:01:51.672297 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:51.672281    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Apr 22 20:01:51.679578 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:51.679555    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-f84sg"]
Apr 22 20:01:51.762177 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:51.762089    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4fc60a74-d84f-49a3-bc71-9059680385f3-certificates\") pod \"keda-operator-ffbb595cb-f84sg\" (UID: \"4fc60a74-d84f-49a3-bc71-9059680385f3\") " pod="openshift-keda/keda-operator-ffbb595cb-f84sg"
Apr 22 20:01:51.762177 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:51.762120    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/4fc60a74-d84f-49a3-bc71-9059680385f3-cabundle0\") pod \"keda-operator-ffbb595cb-f84sg\" (UID: \"4fc60a74-d84f-49a3-bc71-9059680385f3\") " pod="openshift-keda/keda-operator-ffbb595cb-f84sg"
Apr 22 20:01:51.762398 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:51.762217    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zmfm\" (UniqueName: \"kubernetes.io/projected/4fc60a74-d84f-49a3-bc71-9059680385f3-kube-api-access-9zmfm\") pod \"keda-operator-ffbb595cb-f84sg\" (UID: \"4fc60a74-d84f-49a3-bc71-9059680385f3\") " pod="openshift-keda/keda-operator-ffbb595cb-f84sg"
Apr 22 20:01:51.862643 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:51.862611    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9zmfm\" (UniqueName: \"kubernetes.io/projected/4fc60a74-d84f-49a3-bc71-9059680385f3-kube-api-access-9zmfm\") pod \"keda-operator-ffbb595cb-f84sg\" (UID: \"4fc60a74-d84f-49a3-bc71-9059680385f3\") " pod="openshift-keda/keda-operator-ffbb595cb-f84sg"
Apr 22 20:01:51.862836 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:51.862674    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4fc60a74-d84f-49a3-bc71-9059680385f3-certificates\") pod \"keda-operator-ffbb595cb-f84sg\" (UID: \"4fc60a74-d84f-49a3-bc71-9059680385f3\") " pod="openshift-keda/keda-operator-ffbb595cb-f84sg"
Apr 22 20:01:51.862836 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:51.862695    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/4fc60a74-d84f-49a3-bc71-9059680385f3-cabundle0\") pod \"keda-operator-ffbb595cb-f84sg\" (UID: \"4fc60a74-d84f-49a3-bc71-9059680385f3\") " pod="openshift-keda/keda-operator-ffbb595cb-f84sg"
Apr 22 20:01:51.862941 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:51.862833    2576 secret.go:281] references non-existent secret key: ca.crt
Apr 22 20:01:51.862941 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:51.862851    2576 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Apr 22 20:01:51.862941 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:51.862862    2576 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-f84sg: references non-existent secret key: ca.crt
Apr 22 20:01:51.862941 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:51.862925    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fc60a74-d84f-49a3-bc71-9059680385f3-certificates podName:4fc60a74-d84f-49a3-bc71-9059680385f3 nodeName:}" failed. No retries permitted until 2026-04-22 20:01:52.362903879 +0000 UTC m=+247.462860136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4fc60a74-d84f-49a3-bc71-9059680385f3-certificates") pod "keda-operator-ffbb595cb-f84sg" (UID: "4fc60a74-d84f-49a3-bc71-9059680385f3") : references non-existent secret key: ca.crt
Apr 22 20:01:51.863402 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:51.863382    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/4fc60a74-d84f-49a3-bc71-9059680385f3-cabundle0\") pod \"keda-operator-ffbb595cb-f84sg\" (UID: \"4fc60a74-d84f-49a3-bc71-9059680385f3\") " pod="openshift-keda/keda-operator-ffbb595cb-f84sg"
Apr 22 20:01:51.871948 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:51.871922    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zmfm\" (UniqueName: \"kubernetes.io/projected/4fc60a74-d84f-49a3-bc71-9059680385f3-kube-api-access-9zmfm\") pod \"keda-operator-ffbb595cb-f84sg\" (UID: \"4fc60a74-d84f-49a3-bc71-9059680385f3\") " pod="openshift-keda/keda-operator-ffbb595cb-f84sg"
Apr 22 20:01:51.917343 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:51.917270    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg"]
Apr 22 20:01:51.920690 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:51.920672    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg"
Apr 22 20:01:51.923309 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:51.923284    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Apr 22 20:01:51.929853 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:51.929830    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg"]
Apr 22 20:01:52.063795 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:52.063757    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxw9f\" (UniqueName: \"kubernetes.io/projected/05459189-b5c8-40a6-a42c-59a592cbb6ad-kube-api-access-wxw9f\") pod \"keda-metrics-apiserver-7c9f485588-9p5xg\" (UID: \"05459189-b5c8-40a6-a42c-59a592cbb6ad\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg"
Apr 22 20:01:52.063795 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:52.063800    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/05459189-b5c8-40a6-a42c-59a592cbb6ad-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-9p5xg\" (UID: \"05459189-b5c8-40a6-a42c-59a592cbb6ad\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg"
Apr 22 20:01:52.064066 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:52.063863    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/05459189-b5c8-40a6-a42c-59a592cbb6ad-certificates\") pod \"keda-metrics-apiserver-7c9f485588-9p5xg\" (UID: \"05459189-b5c8-40a6-a42c-59a592cbb6ad\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg"
Apr 22 20:01:52.165380 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:52.165340    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/05459189-b5c8-40a6-a42c-59a592cbb6ad-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-9p5xg\" (UID: \"05459189-b5c8-40a6-a42c-59a592cbb6ad\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg"
Apr 22 20:01:52.165569 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:52.165426    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/05459189-b5c8-40a6-a42c-59a592cbb6ad-certificates\") pod \"keda-metrics-apiserver-7c9f485588-9p5xg\" (UID: \"05459189-b5c8-40a6-a42c-59a592cbb6ad\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg"
Apr 22 20:01:52.165569 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:52.165524    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wxw9f\" (UniqueName: \"kubernetes.io/projected/05459189-b5c8-40a6-a42c-59a592cbb6ad-kube-api-access-wxw9f\") pod \"keda-metrics-apiserver-7c9f485588-9p5xg\" (UID: \"05459189-b5c8-40a6-a42c-59a592cbb6ad\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg"
Apr 22 20:01:52.165684 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:52.165606    2576 secret.go:281] references non-existent secret key: tls.crt
Apr 22 20:01:52.165684 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:52.165632    2576 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Apr 22 20:01:52.165684 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:52.165653    2576 projected.go:264] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Apr 22 20:01:52.165684 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:52.165677    2576 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Apr 22 20:01:52.165996 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:52.165718    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/05459189-b5c8-40a6-a42c-59a592cbb6ad-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-9p5xg\" (UID: \"05459189-b5c8-40a6-a42c-59a592cbb6ad\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg"
Apr 22 20:01:52.165996 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:52.165761    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05459189-b5c8-40a6-a42c-59a592cbb6ad-certificates podName:05459189-b5c8-40a6-a42c-59a592cbb6ad nodeName:}" failed. No retries permitted until 2026-04-22 20:01:52.665742584 +0000 UTC m=+247.765698832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/05459189-b5c8-40a6-a42c-59a592cbb6ad-certificates") pod "keda-metrics-apiserver-7c9f485588-9p5xg" (UID: "05459189-b5c8-40a6-a42c-59a592cbb6ad") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Apr 22 20:01:52.174508 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:52.174439    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxw9f\" (UniqueName: \"kubernetes.io/projected/05459189-b5c8-40a6-a42c-59a592cbb6ad-kube-api-access-wxw9f\") pod \"keda-metrics-apiserver-7c9f485588-9p5xg\" (UID: \"05459189-b5c8-40a6-a42c-59a592cbb6ad\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg"
Apr 22 20:01:52.220406 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:52.220374    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-cf49989db-c2rs6"]
Apr 22 20:01:52.223748 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:52.223732    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-c2rs6"
Apr 22 20:01:52.226720 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:52.226698    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Apr 22 20:01:52.231843 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:52.231816    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-c2rs6"]
Apr 22 20:01:52.369576 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:52.369542    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/68824533-a44d-4f61-8e37-ae5624015024-certificates\") pod \"keda-admission-cf49989db-c2rs6\" (UID: \"68824533-a44d-4f61-8e37-ae5624015024\") " pod="openshift-keda/keda-admission-cf49989db-c2rs6"
Apr 22 20:01:52.369986 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:52.369601    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jnvd\" (UniqueName: \"kubernetes.io/projected/68824533-a44d-4f61-8e37-ae5624015024-kube-api-access-9jnvd\") pod \"keda-admission-cf49989db-c2rs6\" (UID: \"68824533-a44d-4f61-8e37-ae5624015024\") " pod="openshift-keda/keda-admission-cf49989db-c2rs6"
Apr 22 20:01:52.369986 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:52.369743    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4fc60a74-d84f-49a3-bc71-9059680385f3-certificates\") pod \"keda-operator-ffbb595cb-f84sg\" (UID: \"4fc60a74-d84f-49a3-bc71-9059680385f3\") " pod="openshift-keda/keda-operator-ffbb595cb-f84sg"
Apr 22 20:01:52.369986 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:52.369859    2576 secret.go:281] references non-existent secret key: ca.crt
Apr 22 20:01:52.369986 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:52.369874    2576 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Apr 22 20:01:52.369986 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:52.369885    2576 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-f84sg: references non-existent secret key: ca.crt
Apr 22 20:01:52.369986 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:52.369967    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fc60a74-d84f-49a3-bc71-9059680385f3-certificates podName:4fc60a74-d84f-49a3-bc71-9059680385f3 nodeName:}" failed. No retries permitted until 2026-04-22 20:01:53.369950145 +0000 UTC m=+248.469906402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4fc60a74-d84f-49a3-bc71-9059680385f3-certificates") pod "keda-operator-ffbb595cb-f84sg" (UID: "4fc60a74-d84f-49a3-bc71-9059680385f3") : references non-existent secret key: ca.crt
Apr 22 20:01:52.470467 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:52.470386    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/68824533-a44d-4f61-8e37-ae5624015024-certificates\") pod \"keda-admission-cf49989db-c2rs6\" (UID: \"68824533-a44d-4f61-8e37-ae5624015024\") " pod="openshift-keda/keda-admission-cf49989db-c2rs6"
Apr 22 20:01:52.470620 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:52.470557    2576 projected.go:264] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Apr 22 20:01:52.470620 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:52.470587    2576 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-cf49989db-c2rs6: secret "keda-admission-webhooks-certs" not found
Apr 22 20:01:52.470716 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:52.470645    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68824533-a44d-4f61-8e37-ae5624015024-certificates podName:68824533-a44d-4f61-8e37-ae5624015024 nodeName:}" failed. No retries permitted until 2026-04-22 20:01:52.970625935 +0000 UTC m=+248.070582184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/68824533-a44d-4f61-8e37-ae5624015024-certificates") pod "keda-admission-cf49989db-c2rs6" (UID: "68824533-a44d-4f61-8e37-ae5624015024") : secret "keda-admission-webhooks-certs" not found
Apr 22 20:01:52.470716 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:52.470555    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9jnvd\" (UniqueName: \"kubernetes.io/projected/68824533-a44d-4f61-8e37-ae5624015024-kube-api-access-9jnvd\") pod \"keda-admission-cf49989db-c2rs6\" (UID: \"68824533-a44d-4f61-8e37-ae5624015024\") " pod="openshift-keda/keda-admission-cf49989db-c2rs6"
Apr 22 20:01:52.481549 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:52.481521    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jnvd\" (UniqueName: \"kubernetes.io/projected/68824533-a44d-4f61-8e37-ae5624015024-kube-api-access-9jnvd\") pod \"keda-admission-cf49989db-c2rs6\" (UID: \"68824533-a44d-4f61-8e37-ae5624015024\") " pod="openshift-keda/keda-admission-cf49989db-c2rs6"
Apr 22 20:01:52.672561 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:52.672531    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/05459189-b5c8-40a6-a42c-59a592cbb6ad-certificates\") pod \"keda-metrics-apiserver-7c9f485588-9p5xg\" (UID: \"05459189-b5c8-40a6-a42c-59a592cbb6ad\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg"
Apr 22 20:01:52.672730 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:52.672697    2576 secret.go:281] references non-existent secret key: tls.crt
Apr 22 20:01:52.672730 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:52.672717    2576 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Apr 22 20:01:52.672855 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:52.672738    2576 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg: references non-existent secret key: tls.crt
Apr 22 20:01:52.672855 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:52.672816    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05459189-b5c8-40a6-a42c-59a592cbb6ad-certificates podName:05459189-b5c8-40a6-a42c-59a592cbb6ad nodeName:}" failed. No retries permitted until 2026-04-22 20:01:53.672803706 +0000 UTC m=+248.772759950 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/05459189-b5c8-40a6-a42c-59a592cbb6ad-certificates") pod "keda-metrics-apiserver-7c9f485588-9p5xg" (UID: "05459189-b5c8-40a6-a42c-59a592cbb6ad") : references non-existent secret key: tls.crt
Apr 22 20:01:52.974914 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:52.974877    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/68824533-a44d-4f61-8e37-ae5624015024-certificates\") pod \"keda-admission-cf49989db-c2rs6\" (UID: \"68824533-a44d-4f61-8e37-ae5624015024\") " pod="openshift-keda/keda-admission-cf49989db-c2rs6"
Apr 22 20:01:52.977628 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:52.977591    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/68824533-a44d-4f61-8e37-ae5624015024-certificates\") pod \"keda-admission-cf49989db-c2rs6\" (UID: \"68824533-a44d-4f61-8e37-ae5624015024\") " pod="openshift-keda/keda-admission-cf49989db-c2rs6"
Apr 22 20:01:53.137899 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:53.137857    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-c2rs6"
Apr 22 20:01:53.258681 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:53.258655    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-c2rs6"]
Apr 22 20:01:53.260751 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:01:53.260718    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68824533_a44d_4f61_8e37_ae5624015024.slice/crio-76490d545a7e605927815a2b16bf3271e255a28c983753f96c60cea8712c9bb6 WatchSource:0}: Error finding container 76490d545a7e605927815a2b16bf3271e255a28c983753f96c60cea8712c9bb6: Status 404 returned error can't find the container with id 76490d545a7e605927815a2b16bf3271e255a28c983753f96c60cea8712c9bb6
Apr 22 20:01:53.366294 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:53.366262    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-c2rs6" event={"ID":"68824533-a44d-4f61-8e37-ae5624015024","Type":"ContainerStarted","Data":"76490d545a7e605927815a2b16bf3271e255a28c983753f96c60cea8712c9bb6"}
Apr 22 20:01:53.379357 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:53.379324    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4fc60a74-d84f-49a3-bc71-9059680385f3-certificates\") pod \"keda-operator-ffbb595cb-f84sg\" (UID: \"4fc60a74-d84f-49a3-bc71-9059680385f3\") " pod="openshift-keda/keda-operator-ffbb595cb-f84sg"
Apr 22 20:01:53.379692 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:53.379483    2576 secret.go:281] references non-existent secret key: ca.crt
Apr 22 20:01:53.379692 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:53.379506    2576 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Apr 22 20:01:53.379692 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:53.379518    2576 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-f84sg: references non-existent secret key: ca.crt
Apr 22 20:01:53.379692 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:53.379576    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fc60a74-d84f-49a3-bc71-9059680385f3-certificates podName:4fc60a74-d84f-49a3-bc71-9059680385f3 nodeName:}" failed. No retries permitted until 2026-04-22 20:01:55.379558347 +0000 UTC m=+250.479514599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4fc60a74-d84f-49a3-bc71-9059680385f3-certificates") pod "keda-operator-ffbb595cb-f84sg" (UID: "4fc60a74-d84f-49a3-bc71-9059680385f3") : references non-existent secret key: ca.crt
Apr 22 20:01:53.682456 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:53.682417    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/05459189-b5c8-40a6-a42c-59a592cbb6ad-certificates\") pod \"keda-metrics-apiserver-7c9f485588-9p5xg\" (UID: \"05459189-b5c8-40a6-a42c-59a592cbb6ad\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg"
Apr 22 20:01:53.682625 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:53.682591    2576 secret.go:281] references non-existent secret key: tls.crt
Apr 22 20:01:53.682625 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:53.682613    2576 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Apr 22 20:01:53.682705 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:53.682636    2576 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg: references non-existent secret key: tls.crt
Apr 22 20:01:53.682705 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:01:53.682700    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05459189-b5c8-40a6-a42c-59a592cbb6ad-certificates podName:05459189-b5c8-40a6-a42c-59a592cbb6ad nodeName:}" failed. No retries permitted until 2026-04-22 20:01:55.68268017 +0000 UTC m=+250.782636417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/05459189-b5c8-40a6-a42c-59a592cbb6ad-certificates") pod "keda-metrics-apiserver-7c9f485588-9p5xg" (UID: "05459189-b5c8-40a6-a42c-59a592cbb6ad") : references non-existent secret key: tls.crt
Apr 22 20:01:55.380580 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:55.380545    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-c2rs6" event={"ID":"68824533-a44d-4f61-8e37-ae5624015024","Type":"ContainerStarted","Data":"bec390becea7113604fd53adae5bfe30d14aeb9d153bcc598e300b2d8f4cf5b1"}
Apr 22 20:01:55.380930 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:55.380636    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-cf49989db-c2rs6"
Apr 22 20:01:55.396872 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:55.396838    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4fc60a74-d84f-49a3-bc71-9059680385f3-certificates\") pod \"keda-operator-ffbb595cb-f84sg\" (UID: \"4fc60a74-d84f-49a3-bc71-9059680385f3\") " pod="openshift-keda/keda-operator-ffbb595cb-f84sg"
Apr 22 20:01:55.399624 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:55.399598    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4fc60a74-d84f-49a3-bc71-9059680385f3-certificates\") pod \"keda-operator-ffbb595cb-f84sg\" (UID: \"4fc60a74-d84f-49a3-bc71-9059680385f3\") " pod="openshift-keda/keda-operator-ffbb595cb-f84sg"
Apr 22 20:01:55.408612 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:55.408571    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-admission-cf49989db-c2rs6" podStartSLOduration=1.83167443 podStartE2EDuration="3.40855945s" podCreationTimestamp="2026-04-22 20:01:52 +0000 UTC" firstStartedPulling="2026-04-22 20:01:53.262047454 +0000 UTC m=+248.362003696" lastFinishedPulling="2026-04-22 20:01:54.838932472 +0000 UTC m=+249.938888716" observedRunningTime="2026-04-22 20:01:55.406352387 +0000 UTC m=+250.506308653" watchObservedRunningTime="2026-04-22 20:01:55.40855945 +0000 UTC m=+250.508515771"
Apr 22 20:01:55.577844 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:55.577809    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-f84sg"
Apr 22 20:01:55.699487 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:55.699447    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/05459189-b5c8-40a6-a42c-59a592cbb6ad-certificates\") pod \"keda-metrics-apiserver-7c9f485588-9p5xg\" (UID: \"05459189-b5c8-40a6-a42c-59a592cbb6ad\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg"
Apr 22 20:01:55.700035 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:55.700015    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-f84sg"]
Apr 22 20:01:55.702198 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:55.702175    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/05459189-b5c8-40a6-a42c-59a592cbb6ad-certificates\") pod \"keda-metrics-apiserver-7c9f485588-9p5xg\" (UID: \"05459189-b5c8-40a6-a42c-59a592cbb6ad\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg"
Apr 22 20:01:55.702724 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:01:55.702700    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc60a74_d84f_49a3_bc71_9059680385f3.slice/crio-9e5b4fa1cc09e12dbb50d58afa0f45c03bb2058b4cb454dc5b4fa6626b951741 WatchSource:0}: Error finding container 9e5b4fa1cc09e12dbb50d58afa0f45c03bb2058b4cb454dc5b4fa6626b951741: Status 404 returned error can't find the container with id 9e5b4fa1cc09e12dbb50d58afa0f45c03bb2058b4cb454dc5b4fa6626b951741
Apr 22 20:01:55.830896 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:55.830869    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg"
Apr 22 20:01:55.954766 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:55.954737    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg"]
Apr 22 20:01:55.957910 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:01:55.957882    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05459189_b5c8_40a6_a42c_59a592cbb6ad.slice/crio-5921ba77c3caf55a479c21a85ad6fec706889d712269e5d05d4ab2d1b29447b7 WatchSource:0}: Error finding container 5921ba77c3caf55a479c21a85ad6fec706889d712269e5d05d4ab2d1b29447b7: Status 404 returned error can't find the container with id 5921ba77c3caf55a479c21a85ad6fec706889d712269e5d05d4ab2d1b29447b7
Apr 22 20:01:56.385508 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:56.385466    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-f84sg" event={"ID":"4fc60a74-d84f-49a3-bc71-9059680385f3","Type":"ContainerStarted","Data":"9e5b4fa1cc09e12dbb50d58afa0f45c03bb2058b4cb454dc5b4fa6626b951741"}
Apr 22 20:01:56.386672 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:56.386640    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg" event={"ID":"05459189-b5c8-40a6-a42c-59a592cbb6ad","Type":"ContainerStarted","Data":"5921ba77c3caf55a479c21a85ad6fec706889d712269e5d05d4ab2d1b29447b7"}
Apr 22 20:01:59.402621 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:59.402586    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-f84sg" event={"ID":"4fc60a74-d84f-49a3-bc71-9059680385f3","Type":"ContainerStarted","Data":"4d1f4be7c0809a3aade920b6959f4341c49aaf9429bffb90213a0c9eabae5111"}
Apr 22 20:01:59.403097 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:59.402882    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-ffbb595cb-f84sg"
Apr 22 20:01:59.420426 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:01:59.420382    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-operator-ffbb595cb-f84sg" podStartSLOduration=5.43573283 podStartE2EDuration="8.420363944s" podCreationTimestamp="2026-04-22 20:01:51 +0000 UTC" firstStartedPulling="2026-04-22 20:01:55.704457673 +0000 UTC m=+250.804413917" lastFinishedPulling="2026-04-22 20:01:58.689088787 +0000 UTC m=+253.789045031" observedRunningTime="2026-04-22 20:01:59.419006559 +0000 UTC m=+254.518962821" watchObservedRunningTime="2026-04-22 20:01:59.420363944 +0000 UTC m=+254.520320209"
Apr 22 20:02:01.410206 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:01.410168    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg" event={"ID":"05459189-b5c8-40a6-a42c-59a592cbb6ad","Type":"ContainerStarted","Data":"94b338750216f5ec211e9031dbcb7ff9cd7aaf530342af5eec4e8009e138f59f"}
Apr 22 20:02:01.410568 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:01.410238    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg"
Apr 22 20:02:01.425995 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:01.425946    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg" podStartSLOduration=5.850533339 podStartE2EDuration="10.42593267s" podCreationTimestamp="2026-04-22 20:01:51 +0000 UTC" firstStartedPulling="2026-04-22 20:01:55.959373818 +0000 UTC m=+251.059330068" lastFinishedPulling="2026-04-22 20:02:00.534773156 +0000 UTC m=+255.634729399" observedRunningTime="2026-04-22 20:02:01.424506878 +0000 UTC m=+256.524463143" watchObservedRunningTime="2026-04-22 20:02:01.42593267 +0000 UTC m=+256.525888935"
Apr 22 20:02:12.364178 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:12.364083    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-k8r4p"
Apr 22 20:02:12.417723 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:12.417696    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-9p5xg"
Apr 22 20:02:16.389168 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:16.389120    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-cf49989db-c2rs6"
Apr 22 20:02:20.408526 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:20.408497    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-ffbb595cb-f84sg"
Apr 22 20:02:45.045592 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.045552    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd"]
Apr 22 20:02:45.049104 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.049087    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd"
Apr 22 20:02:45.052628 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.052604    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4sls\""
Apr 22 20:02:45.052746 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.052606    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Apr 22 20:02:45.053204 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.053188    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Apr 22 20:02:45.057038 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.057017    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd"]
Apr 22 20:02:45.196609 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.196557    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1d54759-117a-40ba-8577-d8ab76767daa-bundle\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd\" (UID: \"a1d54759-117a-40ba-8577-d8ab76767daa\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd"
Apr 22 20:02:45.196609 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.196610    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2gt2\" (UniqueName: \"kubernetes.io/projected/a1d54759-117a-40ba-8577-d8ab76767daa-kube-api-access-x2gt2\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd\" (UID: \"a1d54759-117a-40ba-8577-d8ab76767daa\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd"
Apr 22 20:02:45.196883 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.196634    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1d54759-117a-40ba-8577-d8ab76767daa-util\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd\" (UID: \"a1d54759-117a-40ba-8577-d8ab76767daa\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd"
Apr 22 20:02:45.297225 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.297126    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1d54759-117a-40ba-8577-d8ab76767daa-bundle\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd\" (UID: \"a1d54759-117a-40ba-8577-d8ab76767daa\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd"
Apr 22 20:02:45.297225 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.297195    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x2gt2\" (UniqueName: \"kubernetes.io/projected/a1d54759-117a-40ba-8577-d8ab76767daa-kube-api-access-x2gt2\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd\" (UID: \"a1d54759-117a-40ba-8577-d8ab76767daa\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd"
Apr 22 20:02:45.297444 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.297231    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1d54759-117a-40ba-8577-d8ab76767daa-util\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd\" (UID: \"a1d54759-117a-40ba-8577-d8ab76767daa\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd"
Apr 22 20:02:45.297553 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.297525    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1d54759-117a-40ba-8577-d8ab76767daa-bundle\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd\" (UID: \"a1d54759-117a-40ba-8577-d8ab76767daa\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd"
Apr 22 20:02:45.297642 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.297605    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1d54759-117a-40ba-8577-d8ab76767daa-util\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd\" (UID: \"a1d54759-117a-40ba-8577-d8ab76767daa\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd"
Apr 22 20:02:45.300667 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.300647    2576 kubelet.go:1628] "Image garbage collection succeeded"
Apr 22 20:02:45.306222 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.306203    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Apr 22 20:02:45.317012 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.316994    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Apr 22 20:02:45.327427 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.327399    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2gt2\" (UniqueName: \"kubernetes.io/projected/a1d54759-117a-40ba-8577-d8ab76767daa-kube-api-access-x2gt2\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd\" (UID: \"a1d54759-117a-40ba-8577-d8ab76767daa\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd"
Apr 22 20:02:45.360471 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.360334    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4sls\""
Apr 22 20:02:45.369190 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.369160    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd"
Apr 22 20:02:45.492879 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.492836    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd"]
Apr 22 20:02:45.495832 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:02:45.495803    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d54759_117a_40ba_8577_d8ab76767daa.slice/crio-fbcd4f5ac1e90aeb87f622f8405a696acc9ea0722135842903688aa8f9d53ab3 WatchSource:0}: Error finding container fbcd4f5ac1e90aeb87f622f8405a696acc9ea0722135842903688aa8f9d53ab3: Status 404 returned error can't find the container with id fbcd4f5ac1e90aeb87f622f8405a696acc9ea0722135842903688aa8f9d53ab3
Apr 22 20:02:45.556936 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.556888    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd" event={"ID":"a1d54759-117a-40ba-8577-d8ab76767daa","Type":"ContainerStarted","Data":"074fceeacea81330841f8c9ddc9e6f814a72cebbd9e9c759dd37f148463a45c6"}
Apr 22 20:02:45.557047 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:45.556947    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd" event={"ID":"a1d54759-117a-40ba-8577-d8ab76767daa","Type":"ContainerStarted","Data":"fbcd4f5ac1e90aeb87f622f8405a696acc9ea0722135842903688aa8f9d53ab3"}
Apr 22 20:02:46.561203 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:46.561167    2576 generic.go:358] "Generic (PLEG): container finished" podID="a1d54759-117a-40ba-8577-d8ab76767daa" containerID="074fceeacea81330841f8c9ddc9e6f814a72cebbd9e9c759dd37f148463a45c6" exitCode=0
Apr 22 20:02:46.561203 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:46.561205    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd" event={"ID":"a1d54759-117a-40ba-8577-d8ab76767daa","Type":"ContainerDied","Data":"074fceeacea81330841f8c9ddc9e6f814a72cebbd9e9c759dd37f148463a45c6"}
Apr 22 20:02:46.562305 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:46.562290    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 22 20:02:49.572787 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:49.572689    2576 generic.go:358] "Generic (PLEG): container finished" podID="a1d54759-117a-40ba-8577-d8ab76767daa" containerID="1929844fe60e35e6f315a9e63ee555d682daf0c83d34f923f6c2b087038e4d75" exitCode=0
Apr 22 20:02:49.573125 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:49.572784    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd" event={"ID":"a1d54759-117a-40ba-8577-d8ab76767daa","Type":"ContainerDied","Data":"1929844fe60e35e6f315a9e63ee555d682daf0c83d34f923f6c2b087038e4d75"}
Apr 22 20:02:50.577707 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:50.577676    2576 generic.go:358] "Generic (PLEG): container finished" podID="a1d54759-117a-40ba-8577-d8ab76767daa" containerID="88a13904880c7c845a0edbeea810a4e1a1acebccec99c71370cc9bf3bdc08a0d" exitCode=0
Apr 22 20:02:50.578069 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:50.577782    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd" event={"ID":"a1d54759-117a-40ba-8577-d8ab76767daa","Type":"ContainerDied","Data":"88a13904880c7c845a0edbeea810a4e1a1acebccec99c71370cc9bf3bdc08a0d"}
Apr 22 20:02:51.704197 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:51.704171    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd"
Apr 22 20:02:51.852580 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:51.852470    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1d54759-117a-40ba-8577-d8ab76767daa-util\") pod \"a1d54759-117a-40ba-8577-d8ab76767daa\" (UID: \"a1d54759-117a-40ba-8577-d8ab76767daa\") "
Apr 22 20:02:51.852580 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:51.852528    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1d54759-117a-40ba-8577-d8ab76767daa-bundle\") pod \"a1d54759-117a-40ba-8577-d8ab76767daa\" (UID: \"a1d54759-117a-40ba-8577-d8ab76767daa\") "
Apr 22 20:02:51.852811 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:51.852596    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2gt2\" (UniqueName: \"kubernetes.io/projected/a1d54759-117a-40ba-8577-d8ab76767daa-kube-api-access-x2gt2\") pod \"a1d54759-117a-40ba-8577-d8ab76767daa\" (UID: \"a1d54759-117a-40ba-8577-d8ab76767daa\") "
Apr 22 20:02:51.853236 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:51.853211    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d54759-117a-40ba-8577-d8ab76767daa-bundle" (OuterVolumeSpecName: "bundle") pod "a1d54759-117a-40ba-8577-d8ab76767daa" (UID: "a1d54759-117a-40ba-8577-d8ab76767daa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:02:51.854763 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:51.854741    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d54759-117a-40ba-8577-d8ab76767daa-kube-api-access-x2gt2" (OuterVolumeSpecName: "kube-api-access-x2gt2") pod "a1d54759-117a-40ba-8577-d8ab76767daa" (UID: "a1d54759-117a-40ba-8577-d8ab76767daa"). InnerVolumeSpecName "kube-api-access-x2gt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:02:51.857014 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:51.856988    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d54759-117a-40ba-8577-d8ab76767daa-util" (OuterVolumeSpecName: "util") pod "a1d54759-117a-40ba-8577-d8ab76767daa" (UID: "a1d54759-117a-40ba-8577-d8ab76767daa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:02:51.953409 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:51.953368    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1d54759-117a-40ba-8577-d8ab76767daa-util\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:02:51.953409 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:51.953400    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1d54759-117a-40ba-8577-d8ab76767daa-bundle\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:02:51.953409 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:51.953415    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2gt2\" (UniqueName: \"kubernetes.io/projected/a1d54759-117a-40ba-8577-d8ab76767daa-kube-api-access-x2gt2\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:02:52.586756 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:52.586720    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd" event={"ID":"a1d54759-117a-40ba-8577-d8ab76767daa","Type":"ContainerDied","Data":"fbcd4f5ac1e90aeb87f622f8405a696acc9ea0722135842903688aa8f9d53ab3"}
Apr 22 20:02:52.586756 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:52.586756    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbcd4f5ac1e90aeb87f622f8405a696acc9ea0722135842903688aa8f9d53ab3"
Apr 22 20:02:52.587021 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:52.586848    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dfvfgd"
Apr 22 20:02:57.922190 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:57.922130    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-7qshd"]
Apr 22 20:02:57.922600 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:57.922526    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1d54759-117a-40ba-8577-d8ab76767daa" containerName="pull"
Apr 22 20:02:57.922600 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:57.922541    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1d54759-117a-40ba-8577-d8ab76767daa" containerName="pull"
Apr 22 20:02:57.922600 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:57.922558    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1d54759-117a-40ba-8577-d8ab76767daa" containerName="extract"
Apr 22 20:02:57.922600 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:57.922563    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1d54759-117a-40ba-8577-d8ab76767daa" containerName="extract"
Apr 22 20:02:57.922600 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:57.922580    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1d54759-117a-40ba-8577-d8ab76767daa" containerName="util"
Apr 22 20:02:57.922600 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:57.922585    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1d54759-117a-40ba-8577-d8ab76767daa" containerName="util"
Apr 22 20:02:57.922828 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:57.922632    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="a1d54759-117a-40ba-8577-d8ab76767daa" containerName="extract"
Apr 22 20:02:57.925817 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:57.925800    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-7qshd"
Apr 22 20:02:57.928851 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:57.928829    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Apr 22 20:02:57.928958 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:57.928938    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-mn57v\""
Apr 22 20:02:57.929211 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:57.929194    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Apr 22 20:02:57.937494 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:57.937461    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-7qshd"]
Apr 22 20:02:58.009980 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:58.009941    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-758cp\" (UniqueName: \"kubernetes.io/projected/0e9130cb-b3c3-4e38-bcb2-8e5eacb35065-kube-api-access-758cp\") pod \"cert-manager-operator-controller-manager-54b9655956-7qshd\" (UID: \"0e9130cb-b3c3-4e38-bcb2-8e5eacb35065\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-7qshd"
Apr 22 20:02:58.010134 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:58.009994    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e9130cb-b3c3-4e38-bcb2-8e5eacb35065-tmp\") pod \"cert-manager-operator-controller-manager-54b9655956-7qshd\" (UID: \"0e9130cb-b3c3-4e38-bcb2-8e5eacb35065\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-7qshd"
Apr 22 20:02:58.111417 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:58.111382    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-758cp\" (UniqueName: \"kubernetes.io/projected/0e9130cb-b3c3-4e38-bcb2-8e5eacb35065-kube-api-access-758cp\") pod \"cert-manager-operator-controller-manager-54b9655956-7qshd\" (UID: \"0e9130cb-b3c3-4e38-bcb2-8e5eacb35065\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-7qshd"
Apr 22 20:02:58.111626 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:58.111440    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e9130cb-b3c3-4e38-bcb2-8e5eacb35065-tmp\") pod \"cert-manager-operator-controller-manager-54b9655956-7qshd\" (UID: \"0e9130cb-b3c3-4e38-bcb2-8e5eacb35065\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-7qshd"
Apr 22 20:02:58.111780 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:58.111763    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e9130cb-b3c3-4e38-bcb2-8e5eacb35065-tmp\") pod \"cert-manager-operator-controller-manager-54b9655956-7qshd\" (UID: \"0e9130cb-b3c3-4e38-bcb2-8e5eacb35065\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-7qshd"
Apr 22 20:02:58.119849 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:58.119828    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-758cp\" (UniqueName: \"kubernetes.io/projected/0e9130cb-b3c3-4e38-bcb2-8e5eacb35065-kube-api-access-758cp\") pod \"cert-manager-operator-controller-manager-54b9655956-7qshd\" (UID: \"0e9130cb-b3c3-4e38-bcb2-8e5eacb35065\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-7qshd"
Apr 22 20:02:58.235063 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:58.234975    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-7qshd"
Apr 22 20:02:58.380670 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:58.380631    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-7qshd"]
Apr 22 20:02:58.383832 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:02:58.383802    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9130cb_b3c3_4e38_bcb2_8e5eacb35065.slice/crio-c0cab329e9b0cf44cce1e4f376cbddfb11915ff1a8112125af2a5118c475029f WatchSource:0}: Error finding container c0cab329e9b0cf44cce1e4f376cbddfb11915ff1a8112125af2a5118c475029f: Status 404 returned error can't find the container with id c0cab329e9b0cf44cce1e4f376cbddfb11915ff1a8112125af2a5118c475029f
Apr 22 20:02:58.606378 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:02:58.606296    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-7qshd" event={"ID":"0e9130cb-b3c3-4e38-bcb2-8e5eacb35065","Type":"ContainerStarted","Data":"c0cab329e9b0cf44cce1e4f376cbddfb11915ff1a8112125af2a5118c475029f"}
Apr 22 20:03:00.615160 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:00.615107    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-7qshd" event={"ID":"0e9130cb-b3c3-4e38-bcb2-8e5eacb35065","Type":"ContainerStarted","Data":"e1944feb6da267c01171b1280c3d0e1497d2c767f49f4012d47132c8ef2d7a94"}
Apr 22 20:03:07.376894 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:07.376828    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-7qshd" podStartSLOduration=9.126463837 podStartE2EDuration="10.376811641s" podCreationTimestamp="2026-04-22 20:02:57 +0000 UTC" firstStartedPulling="2026-04-22 20:02:58.386431612 +0000 UTC m=+313.486387855" lastFinishedPulling="2026-04-22 20:02:59.636779413 +0000 UTC m=+314.736735659" observedRunningTime="2026-04-22 20:03:00.641857056 +0000 UTC m=+315.741813320" watchObservedRunningTime="2026-04-22 20:03:07.376811641 +0000 UTC m=+322.476767910"
Apr 22 20:03:07.377925 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:07.377906    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt"]
Apr 22 20:03:07.381438 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:07.381420    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt"
Apr 22 20:03:07.383777 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:07.383760    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Apr 22 20:03:07.384488 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:07.384471    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Apr 22 20:03:07.384576 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:07.384476    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4sls\""
Apr 22 20:03:07.387839 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:07.387818    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt"]
Apr 22 20:03:07.488338 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:07.488300    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkwnd\" (UniqueName: \"kubernetes.io/projected/783907ab-5728-4ba1-93d7-9398c5a38b20-kube-api-access-lkwnd\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt\" (UID: \"783907ab-5728-4ba1-93d7-9398c5a38b20\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt"
Apr 22 20:03:07.488494 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:07.488339    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/783907ab-5728-4ba1-93d7-9398c5a38b20-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt\" (UID: \"783907ab-5728-4ba1-93d7-9398c5a38b20\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt"
Apr 22 20:03:07.488494 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:07.488425    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/783907ab-5728-4ba1-93d7-9398c5a38b20-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt\" (UID: \"783907ab-5728-4ba1-93d7-9398c5a38b20\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt"
Apr 22 20:03:07.589378 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:07.589340    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/783907ab-5728-4ba1-93d7-9398c5a38b20-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt\" (UID: \"783907ab-5728-4ba1-93d7-9398c5a38b20\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt"
Apr 22 20:03:07.589592 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:07.589442    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lkwnd\" (UniqueName: \"kubernetes.io/projected/783907ab-5728-4ba1-93d7-9398c5a38b20-kube-api-access-lkwnd\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt\" (UID: \"783907ab-5728-4ba1-93d7-9398c5a38b20\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt"
Apr 22 20:03:07.589592 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:07.589474    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/783907ab-5728-4ba1-93d7-9398c5a38b20-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt\" (UID: \"783907ab-5728-4ba1-93d7-9398c5a38b20\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt"
Apr 22 20:03:07.589750 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:07.589731    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/783907ab-5728-4ba1-93d7-9398c5a38b20-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt\" (UID: \"783907ab-5728-4ba1-93d7-9398c5a38b20\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt"
Apr 22 20:03:07.589813 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:07.589764    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/783907ab-5728-4ba1-93d7-9398c5a38b20-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt\" (UID: \"783907ab-5728-4ba1-93d7-9398c5a38b20\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt"
Apr 22 20:03:07.597545 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:07.597522    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkwnd\" (UniqueName: \"kubernetes.io/projected/783907ab-5728-4ba1-93d7-9398c5a38b20-kube-api-access-lkwnd\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt\" (UID: \"783907ab-5728-4ba1-93d7-9398c5a38b20\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt"
Apr 22 20:03:07.692073 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:07.691987    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt"
Apr 22 20:03:07.817716 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:07.817692    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt"]
Apr 22 20:03:07.820242 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:03:07.820210    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783907ab_5728_4ba1_93d7_9398c5a38b20.slice/crio-2190ffe2dcbbf795d371c4c9e77e214d8ed600163783955e9930472c5b37a520 WatchSource:0}: Error finding container 2190ffe2dcbbf795d371c4c9e77e214d8ed600163783955e9930472c5b37a520: Status 404 returned error can't find the container with id 2190ffe2dcbbf795d371c4c9e77e214d8ed600163783955e9930472c5b37a520
Apr 22 20:03:08.645536 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:08.645502    2576 generic.go:358] "Generic (PLEG): container finished" podID="783907ab-5728-4ba1-93d7-9398c5a38b20" containerID="f0681ca67584e5e57061309633a1aa51c9c8ec89d76707ef640ae41d3373faeb" exitCode=0
Apr 22 20:03:08.645977 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:08.645554    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt" event={"ID":"783907ab-5728-4ba1-93d7-9398c5a38b20","Type":"ContainerDied","Data":"f0681ca67584e5e57061309633a1aa51c9c8ec89d76707ef640ae41d3373faeb"}
Apr 22 20:03:08.645977 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:08.645581    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt" event={"ID":"783907ab-5728-4ba1-93d7-9398c5a38b20","Type":"ContainerStarted","Data":"2190ffe2dcbbf795d371c4c9e77e214d8ed600163783955e9930472c5b37a520"}
Apr 22 20:03:10.654306 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:10.654269    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt" event={"ID":"783907ab-5728-4ba1-93d7-9398c5a38b20","Type":"ContainerStarted","Data":"f80413e56545779971f57149ebe2a797a513ebeded57a1f53fa76902a6ba84f8"}
Apr 22 20:03:11.659318 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:11.659286    2576 generic.go:358] "Generic (PLEG): container finished" podID="783907ab-5728-4ba1-93d7-9398c5a38b20" containerID="f80413e56545779971f57149ebe2a797a513ebeded57a1f53fa76902a6ba84f8" exitCode=0
Apr 22 20:03:11.659717 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:11.659363    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt" event={"ID":"783907ab-5728-4ba1-93d7-9398c5a38b20","Type":"ContainerDied","Data":"f80413e56545779971f57149ebe2a797a513ebeded57a1f53fa76902a6ba84f8"}
Apr 22 20:03:12.664075 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:12.664039    2576 generic.go:358] "Generic (PLEG): container finished" podID="783907ab-5728-4ba1-93d7-9398c5a38b20" containerID="d78da97bfae430db8c8e5c6c8390b7586b273697b14fe7ea2718132b9186ad38" exitCode=0
Apr 22 20:03:12.664471 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:12.664131    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt" event={"ID":"783907ab-5728-4ba1-93d7-9398c5a38b20","Type":"ContainerDied","Data":"d78da97bfae430db8c8e5c6c8390b7586b273697b14fe7ea2718132b9186ad38"}
Apr 22 20:03:13.787568 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:13.787540    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt"
Apr 22 20:03:13.845723 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:13.845688    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkwnd\" (UniqueName: \"kubernetes.io/projected/783907ab-5728-4ba1-93d7-9398c5a38b20-kube-api-access-lkwnd\") pod \"783907ab-5728-4ba1-93d7-9398c5a38b20\" (UID: \"783907ab-5728-4ba1-93d7-9398c5a38b20\") "
Apr 22 20:03:13.845862 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:13.845738    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/783907ab-5728-4ba1-93d7-9398c5a38b20-bundle\") pod \"783907ab-5728-4ba1-93d7-9398c5a38b20\" (UID: \"783907ab-5728-4ba1-93d7-9398c5a38b20\") "
Apr 22 20:03:13.845862 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:13.845764    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/783907ab-5728-4ba1-93d7-9398c5a38b20-util\") pod \"783907ab-5728-4ba1-93d7-9398c5a38b20\" (UID: \"783907ab-5728-4ba1-93d7-9398c5a38b20\") "
Apr 22 20:03:13.846180 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:13.846136    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783907ab-5728-4ba1-93d7-9398c5a38b20-bundle" (OuterVolumeSpecName: "bundle") pod "783907ab-5728-4ba1-93d7-9398c5a38b20" (UID: "783907ab-5728-4ba1-93d7-9398c5a38b20"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:03:13.847984 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:13.847958    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783907ab-5728-4ba1-93d7-9398c5a38b20-kube-api-access-lkwnd" (OuterVolumeSpecName: "kube-api-access-lkwnd") pod "783907ab-5728-4ba1-93d7-9398c5a38b20" (UID: "783907ab-5728-4ba1-93d7-9398c5a38b20"). InnerVolumeSpecName "kube-api-access-lkwnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:03:13.850491 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:13.850474    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783907ab-5728-4ba1-93d7-9398c5a38b20-util" (OuterVolumeSpecName: "util") pod "783907ab-5728-4ba1-93d7-9398c5a38b20" (UID: "783907ab-5728-4ba1-93d7-9398c5a38b20"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:03:13.946948 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:13.946874    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkwnd\" (UniqueName: \"kubernetes.io/projected/783907ab-5728-4ba1-93d7-9398c5a38b20-kube-api-access-lkwnd\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:03:13.946948 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:13.946903    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/783907ab-5728-4ba1-93d7-9398c5a38b20-bundle\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:03:13.946948 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:13.946914    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/783907ab-5728-4ba1-93d7-9398c5a38b20-util\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:03:14.676119 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:14.676083    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt" event={"ID":"783907ab-5728-4ba1-93d7-9398c5a38b20","Type":"ContainerDied","Data":"2190ffe2dcbbf795d371c4c9e77e214d8ed600163783955e9930472c5b37a520"}
Apr 22 20:03:14.676119 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:14.676117    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2190ffe2dcbbf795d371c4c9e77e214d8ed600163783955e9930472c5b37a520"
Apr 22 20:03:14.676351 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:14.676142    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxvngt"
Apr 22 20:03:36.978792 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:36.978713    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98"]
Apr 22 20:03:36.979207 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:36.979021    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="783907ab-5728-4ba1-93d7-9398c5a38b20" containerName="extract"
Apr 22 20:03:36.979207 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:36.979031    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="783907ab-5728-4ba1-93d7-9398c5a38b20" containerName="extract"
Apr 22 20:03:36.979207 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:36.979044    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="783907ab-5728-4ba1-93d7-9398c5a38b20" containerName="util"
Apr 22 20:03:36.979207 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:36.979050    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="783907ab-5728-4ba1-93d7-9398c5a38b20" containerName="util"
Apr 22 20:03:36.979207 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:36.979059    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="783907ab-5728-4ba1-93d7-9398c5a38b20" containerName="pull"
Apr 22 20:03:36.979207 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:36.979064    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="783907ab-5728-4ba1-93d7-9398c5a38b20" containerName="pull"
Apr 22 20:03:36.979207 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:36.979117    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="783907ab-5728-4ba1-93d7-9398c5a38b20" containerName="extract"
Apr 22 20:03:36.982044 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:36.982024    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98"
Apr 22 20:03:36.984108 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:36.984080    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Apr 22 20:03:36.984250 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:36.984189    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Apr 22 20:03:36.984884 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:36.984865    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4sls\""
Apr 22 20:03:36.988200 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:36.988180    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98"]
Apr 22 20:03:37.159355 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:37.159320    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0-bundle\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98\" (UID: \"b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98"
Apr 22 20:03:37.159522 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:37.159359    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2fns\" (UniqueName: \"kubernetes.io/projected/b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0-kube-api-access-g2fns\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98\" (UID: \"b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98"
Apr 22 20:03:37.159522 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:37.159385    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0-util\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98\" (UID: \"b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98"
Apr 22 20:03:37.260249 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:37.260144    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g2fns\" (UniqueName: \"kubernetes.io/projected/b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0-kube-api-access-g2fns\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98\" (UID: \"b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98"
Apr 22 20:03:37.260249 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:37.260202    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0-util\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98\" (UID: \"b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98"
Apr 22 20:03:37.260464 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:37.260271    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0-bundle\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98\" (UID: \"b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98"
Apr 22 20:03:37.260567 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:37.260549    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0-bundle\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98\" (UID: \"b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98"
Apr 22 20:03:37.260649 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:37.260625    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0-util\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98\" (UID: \"b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98"
Apr 22 20:03:37.268116 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:37.268095    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2fns\" (UniqueName: \"kubernetes.io/projected/b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0-kube-api-access-g2fns\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98\" (UID: \"b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98"
Apr 22 20:03:37.292116 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:37.292093    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98"
Apr 22 20:03:37.422403 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:37.422377    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98"]
Apr 22 20:03:37.424849 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:03:37.424816    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44bcbf6_2f5b_4e2c_b497_fd797d3ed7f0.slice/crio-05a71e6221477f8282a78e8574be3d5e6926a2b85e93d88fa56eb9a503d14f71 WatchSource:0}: Error finding container 05a71e6221477f8282a78e8574be3d5e6926a2b85e93d88fa56eb9a503d14f71: Status 404 returned error can't find the container with id 05a71e6221477f8282a78e8574be3d5e6926a2b85e93d88fa56eb9a503d14f71
Apr 22 20:03:37.757627 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:37.757587    2576 generic.go:358] "Generic (PLEG): container finished" podID="b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0" containerID="b4faa143cd5ed9956fd5852796e2d6b34fee5f9e72fc846cdf8e54afea034c13" exitCode=0
Apr 22 20:03:37.758048 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:37.757676    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98" event={"ID":"b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0","Type":"ContainerDied","Data":"b4faa143cd5ed9956fd5852796e2d6b34fee5f9e72fc846cdf8e54afea034c13"}
Apr 22 20:03:37.758048 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:37.757719    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98" event={"ID":"b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0","Type":"ContainerStarted","Data":"05a71e6221477f8282a78e8574be3d5e6926a2b85e93d88fa56eb9a503d14f71"}
Apr 22 20:03:38.762334 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:38.762300    2576 generic.go:358] "Generic (PLEG): container finished" podID="b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0" containerID="bb4a7230f4d48e41f39e37455922db0ce6ce3a9e30375d1ff1de73609096dd93" exitCode=0
Apr 22 20:03:38.762751 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:38.762353    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98" event={"ID":"b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0","Type":"ContainerDied","Data":"bb4a7230f4d48e41f39e37455922db0ce6ce3a9e30375d1ff1de73609096dd93"}
Apr 22 20:03:39.767587 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:39.767540    2576 generic.go:358] "Generic (PLEG): container finished" podID="b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0" containerID="16930a2ef21d0423861440041b66010003f1c7444f7edd5d26a04f2750a2bf38" exitCode=0
Apr 22 20:03:39.767587 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:39.767590    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98" event={"ID":"b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0","Type":"ContainerDied","Data":"16930a2ef21d0423861440041b66010003f1c7444f7edd5d26a04f2750a2bf38"}
Apr 22 20:03:40.896163 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:40.896130    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98"
Apr 22 20:03:40.990280 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:40.990243    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2fns\" (UniqueName: \"kubernetes.io/projected/b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0-kube-api-access-g2fns\") pod \"b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0\" (UID: \"b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0\") "
Apr 22 20:03:40.990446 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:40.990309    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0-util\") pod \"b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0\" (UID: \"b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0\") "
Apr 22 20:03:40.990446 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:40.990373    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0-bundle\") pod \"b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0\" (UID: \"b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0\") "
Apr 22 20:03:40.991141 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:40.991114    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0-bundle" (OuterVolumeSpecName: "bundle") pod "b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0" (UID: "b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:03:40.992437 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:40.992414    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0-kube-api-access-g2fns" (OuterVolumeSpecName: "kube-api-access-g2fns") pod "b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0" (UID: "b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0"). InnerVolumeSpecName "kube-api-access-g2fns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:03:40.995421 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:40.995400    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0-util" (OuterVolumeSpecName: "util") pod "b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0" (UID: "b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:03:41.091587 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:41.091496    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2fns\" (UniqueName: \"kubernetes.io/projected/b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0-kube-api-access-g2fns\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:03:41.091587 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:41.091533    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0-util\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:03:41.091587 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:41.091543    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0-bundle\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:03:41.776019 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:41.775937    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98"
Apr 22 20:03:41.776019 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:41.775945    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835whv98" event={"ID":"b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0","Type":"ContainerDied","Data":"05a71e6221477f8282a78e8574be3d5e6926a2b85e93d88fa56eb9a503d14f71"}
Apr 22 20:03:41.776019 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:41.775973    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05a71e6221477f8282a78e8574be3d5e6926a2b85e93d88fa56eb9a503d14f71"
Apr 22 20:03:46.919027 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:46.918990    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv"]
Apr 22 20:03:46.919416 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:46.919344    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0" containerName="extract"
Apr 22 20:03:46.919416 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:46.919358    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0" containerName="extract"
Apr 22 20:03:46.919416 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:46.919372    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0" containerName="util"
Apr 22 20:03:46.919416 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:46.919377    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0" containerName="util"
Apr 22 20:03:46.919416 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:46.919385    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0" containerName="pull"
Apr 22 20:03:46.919416 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:46.919390    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0" containerName="pull"
Apr 22 20:03:46.919590 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:46.919461    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="b44bcbf6-2f5b-4e2c-b497-fd797d3ed7f0" containerName="extract"
Apr 22 20:03:46.923684 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:46.923666    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv"
Apr 22 20:03:46.927141 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:46.927120    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Apr 22 20:03:46.928077 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:46.928058    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Apr 22 20:03:46.928130 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:46.928113    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4sls\""
Apr 22 20:03:46.937018 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:46.936993    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv"]
Apr 22 20:03:46.939544 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:46.939522    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/78538159-e21f-4583-9183-0be41f6613a9-bundle\") pod \"7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv\" (UID: \"78538159-e21f-4583-9183-0be41f6613a9\") " pod="openshift-marketplace/7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv"
Apr 22 20:03:46.939661 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:46.939549    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/78538159-e21f-4583-9183-0be41f6613a9-util\") pod \"7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv\" (UID: \"78538159-e21f-4583-9183-0be41f6613a9\") " pod="openshift-marketplace/7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv"
Apr 22 20:03:46.939661 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:46.939588    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6x5v\" (UniqueName: \"kubernetes.io/projected/78538159-e21f-4583-9183-0be41f6613a9-kube-api-access-c6x5v\") pod \"7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv\" (UID: \"78538159-e21f-4583-9183-0be41f6613a9\") " pod="openshift-marketplace/7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv"
Apr 22 20:03:47.040117 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:47.040079    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/78538159-e21f-4583-9183-0be41f6613a9-bundle\") pod \"7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv\" (UID: \"78538159-e21f-4583-9183-0be41f6613a9\") " pod="openshift-marketplace/7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv"
Apr 22 20:03:47.040117 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:47.040118    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/78538159-e21f-4583-9183-0be41f6613a9-util\") pod \"7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv\" (UID: \"78538159-e21f-4583-9183-0be41f6613a9\") " pod="openshift-marketplace/7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv"
Apr 22 20:03:47.040354 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:47.040190    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c6x5v\" (UniqueName: \"kubernetes.io/projected/78538159-e21f-4583-9183-0be41f6613a9-kube-api-access-c6x5v\") pod \"7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv\" (UID: \"78538159-e21f-4583-9183-0be41f6613a9\") " pod="openshift-marketplace/7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv"
Apr 22 20:03:47.040487 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:47.040464    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/78538159-e21f-4583-9183-0be41f6613a9-bundle\") pod \"7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv\" (UID: \"78538159-e21f-4583-9183-0be41f6613a9\") " pod="openshift-marketplace/7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv"
Apr 22 20:03:47.040536 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:47.040491    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/78538159-e21f-4583-9183-0be41f6613a9-util\") pod \"7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv\" (UID: \"78538159-e21f-4583-9183-0be41f6613a9\") " pod="openshift-marketplace/7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv"
Apr 22 20:03:47.066666 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:47.066644    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6x5v\" (UniqueName: \"kubernetes.io/projected/78538159-e21f-4583-9183-0be41f6613a9-kube-api-access-c6x5v\") pod \"7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv\" (UID: \"78538159-e21f-4583-9183-0be41f6613a9\") " pod="openshift-marketplace/7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv"
Apr 22 20:03:47.234322 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:47.234228    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv"
Apr 22 20:03:47.388511 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:47.388482    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv"]
Apr 22 20:03:47.394213 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:03:47.394180    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78538159_e21f_4583_9183_0be41f6613a9.slice/crio-875409ea72ac6d786f4825b96d572df8520b281b188669875997172f5187aa85 WatchSource:0}: Error finding container 875409ea72ac6d786f4825b96d572df8520b281b188669875997172f5187aa85: Status 404 returned error can't find the container with id 875409ea72ac6d786f4825b96d572df8520b281b188669875997172f5187aa85
Apr 22 20:03:47.807481 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:47.807390    2576 generic.go:358] "Generic (PLEG): container finished" podID="78538159-e21f-4583-9183-0be41f6613a9" containerID="a5ac71c8c6e9d299e45b3dea67450ef43639b167e08aee9d746183fc23c9ebe7" exitCode=0
Apr 22 20:03:47.807481 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:47.807450    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv" event={"ID":"78538159-e21f-4583-9183-0be41f6613a9","Type":"ContainerDied","Data":"a5ac71c8c6e9d299e45b3dea67450ef43639b167e08aee9d746183fc23c9ebe7"}
Apr 22 20:03:47.807481 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:47.807479    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv" event={"ID":"78538159-e21f-4583-9183-0be41f6613a9","Type":"ContainerStarted","Data":"875409ea72ac6d786f4825b96d572df8520b281b188669875997172f5187aa85"}
Apr 22 20:03:48.197526 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:48.197475    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-zr8xg"]
Apr 22 20:03:48.200805 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:48.200788    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators/servicemesh-operator3-55f49c5f94-zr8xg"
Apr 22 20:03:48.203765 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:48.203742    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-operators\"/\"servicemesh-operator3-dockercfg-dkqf8\""
Apr 22 20:03:48.203876 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:48.203742    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators\"/\"openshift-service-ca.crt\""
Apr 22 20:03:48.203876 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:48.203798    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators\"/\"kube-root-ca.crt\""
Apr 22 20:03:48.214844 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:48.214820    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-zr8xg"]
Apr 22 20:03:48.248288 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:48.248259    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwbvl\" (UniqueName: \"kubernetes.io/projected/cee9627d-3940-4aa1-be31-94d1076c54a8-kube-api-access-gwbvl\") pod \"servicemesh-operator3-55f49c5f94-zr8xg\" (UID: \"cee9627d-3940-4aa1-be31-94d1076c54a8\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-zr8xg"
Apr 22 20:03:48.248429 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:48.248296    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/cee9627d-3940-4aa1-be31-94d1076c54a8-operator-config\") pod \"servicemesh-operator3-55f49c5f94-zr8xg\" (UID: \"cee9627d-3940-4aa1-be31-94d1076c54a8\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-zr8xg"
Apr 22 20:03:48.349058 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:48.348983    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gwbvl\" (UniqueName: \"kubernetes.io/projected/cee9627d-3940-4aa1-be31-94d1076c54a8-kube-api-access-gwbvl\") pod \"servicemesh-operator3-55f49c5f94-zr8xg\" (UID: \"cee9627d-3940-4aa1-be31-94d1076c54a8\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-zr8xg"
Apr 22 20:03:48.349058 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:48.349022    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/cee9627d-3940-4aa1-be31-94d1076c54a8-operator-config\") pod \"servicemesh-operator3-55f49c5f94-zr8xg\" (UID: \"cee9627d-3940-4aa1-be31-94d1076c54a8\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-zr8xg"
Apr 22 20:03:48.351553 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:48.351515    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/cee9627d-3940-4aa1-be31-94d1076c54a8-operator-config\") pod \"servicemesh-operator3-55f49c5f94-zr8xg\" (UID: \"cee9627d-3940-4aa1-be31-94d1076c54a8\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-zr8xg"
Apr 22 20:03:48.361916 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:48.361885    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwbvl\" (UniqueName: \"kubernetes.io/projected/cee9627d-3940-4aa1-be31-94d1076c54a8-kube-api-access-gwbvl\") pod \"servicemesh-operator3-55f49c5f94-zr8xg\" (UID: \"cee9627d-3940-4aa1-be31-94d1076c54a8\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-zr8xg"
Apr 22 20:03:48.509210 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:48.509186    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators/servicemesh-operator3-55f49c5f94-zr8xg"
Apr 22 20:03:48.647667 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:48.647631    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-zr8xg"]
Apr 22 20:03:48.651449 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:03:48.651425    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee9627d_3940_4aa1_be31_94d1076c54a8.slice/crio-cf066c4160512e174bfbd8dc53cd352624fd52b7378e70454c66ef6e5c494189 WatchSource:0}: Error finding container cf066c4160512e174bfbd8dc53cd352624fd52b7378e70454c66ef6e5c494189: Status 404 returned error can't find the container with id cf066c4160512e174bfbd8dc53cd352624fd52b7378e70454c66ef6e5c494189
Apr 22 20:03:48.811681 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:48.811646    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-operators/servicemesh-operator3-55f49c5f94-zr8xg" event={"ID":"cee9627d-3940-4aa1-be31-94d1076c54a8","Type":"ContainerStarted","Data":"cf066c4160512e174bfbd8dc53cd352624fd52b7378e70454c66ef6e5c494189"}
Apr 22 20:03:48.813182 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:48.813141    2576 generic.go:358] "Generic (PLEG): container finished" podID="78538159-e21f-4583-9183-0be41f6613a9" containerID="5d6673910cc206b9f0a6e3f77ffc08bcfb43262b051036101b97c7043f69e475" exitCode=0
Apr 22 20:03:48.813276 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:48.813216    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv" event={"ID":"78538159-e21f-4583-9183-0be41f6613a9","Type":"ContainerDied","Data":"5d6673910cc206b9f0a6e3f77ffc08bcfb43262b051036101b97c7043f69e475"}
Apr 22 20:03:49.818613 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:49.818583    2576 generic.go:358] "Generic (PLEG): container finished" podID="78538159-e21f-4583-9183-0be41f6613a9" containerID="2f02299bcd93601a38b89cce57952d331463055965238f529bef9aa42ffec667" exitCode=0
Apr 22 20:03:49.818998 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:49.818674    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv" event={"ID":"78538159-e21f-4583-9183-0be41f6613a9","Type":"ContainerDied","Data":"2f02299bcd93601a38b89cce57952d331463055965238f529bef9aa42ffec667"}
Apr 22 20:03:50.955904 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:50.955878    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv"
Apr 22 20:03:50.968380 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:50.968346    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/78538159-e21f-4583-9183-0be41f6613a9-bundle\") pod \"78538159-e21f-4583-9183-0be41f6613a9\" (UID: \"78538159-e21f-4583-9183-0be41f6613a9\") "
Apr 22 20:03:50.968555 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:50.968408    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6x5v\" (UniqueName: \"kubernetes.io/projected/78538159-e21f-4583-9183-0be41f6613a9-kube-api-access-c6x5v\") pod \"78538159-e21f-4583-9183-0be41f6613a9\" (UID: \"78538159-e21f-4583-9183-0be41f6613a9\") "
Apr 22 20:03:50.969489 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:50.969447    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78538159-e21f-4583-9183-0be41f6613a9-bundle" (OuterVolumeSpecName: "bundle") pod "78538159-e21f-4583-9183-0be41f6613a9" (UID: "78538159-e21f-4583-9183-0be41f6613a9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:03:50.971009 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:50.970971    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78538159-e21f-4583-9183-0be41f6613a9-kube-api-access-c6x5v" (OuterVolumeSpecName: "kube-api-access-c6x5v") pod "78538159-e21f-4583-9183-0be41f6613a9" (UID: "78538159-e21f-4583-9183-0be41f6613a9"). InnerVolumeSpecName "kube-api-access-c6x5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:03:51.069320 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:51.069286    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/78538159-e21f-4583-9183-0be41f6613a9-util\") pod \"78538159-e21f-4583-9183-0be41f6613a9\" (UID: \"78538159-e21f-4583-9183-0be41f6613a9\") "
Apr 22 20:03:51.069523 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:51.069446    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/78538159-e21f-4583-9183-0be41f6613a9-bundle\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:03:51.069523 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:51.069468    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6x5v\" (UniqueName: \"kubernetes.io/projected/78538159-e21f-4583-9183-0be41f6613a9-kube-api-access-c6x5v\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:03:51.075004 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:51.074921    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78538159-e21f-4583-9183-0be41f6613a9-util" (OuterVolumeSpecName: "util") pod "78538159-e21f-4583-9183-0be41f6613a9" (UID: "78538159-e21f-4583-9183-0be41f6613a9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:03:51.170339 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:51.170303    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/78538159-e21f-4583-9183-0be41f6613a9-util\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:03:51.828024 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:51.827946    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv"
Apr 22 20:03:51.828024 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:51.827950    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/7797e60c2c2aaccf623f93d365c9b5c5cd3662e5f903c80e749ff805ebxvswv" event={"ID":"78538159-e21f-4583-9183-0be41f6613a9","Type":"ContainerDied","Data":"875409ea72ac6d786f4825b96d572df8520b281b188669875997172f5187aa85"}
Apr 22 20:03:51.828024 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:51.827990    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="875409ea72ac6d786f4825b96d572df8520b281b188669875997172f5187aa85"
Apr 22 20:03:52.834135 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:52.834095    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-operators/servicemesh-operator3-55f49c5f94-zr8xg" event={"ID":"cee9627d-3940-4aa1-be31-94d1076c54a8","Type":"ContainerStarted","Data":"cc2bb493e60edba1d0a94b70a0d5b8eec4fca39bf5ca49fd80418397223914b0"}
Apr 22 20:03:52.834563 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:52.834185    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-operators/servicemesh-operator3-55f49c5f94-zr8xg"
Apr 22 20:03:52.854028 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:52.853961    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-operators/servicemesh-operator3-55f49c5f94-zr8xg" podStartSLOduration=1.483075957 podStartE2EDuration="4.853945725s" podCreationTimestamp="2026-04-22 20:03:48 +0000 UTC" firstStartedPulling="2026-04-22 20:03:48.653823734 +0000 UTC m=+363.753779977" lastFinishedPulling="2026-04-22 20:03:52.024693499 +0000 UTC m=+367.124649745" observedRunningTime="2026-04-22 20:03:52.852822427 +0000 UTC m=+367.952778703" watchObservedRunningTime="2026-04-22 20:03:52.853945725 +0000 UTC m=+367.953901989"
Apr 22 20:03:53.266400 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.266361    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"]
Apr 22 20:03:53.266717 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.266704    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78538159-e21f-4583-9183-0be41f6613a9" containerName="util"
Apr 22 20:03:53.266766 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.266719    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="78538159-e21f-4583-9183-0be41f6613a9" containerName="util"
Apr 22 20:03:53.266766 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.266729    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78538159-e21f-4583-9183-0be41f6613a9" containerName="pull"
Apr 22 20:03:53.266766 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.266734    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="78538159-e21f-4583-9183-0be41f6613a9" containerName="pull"
Apr 22 20:03:53.266766 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.266744    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78538159-e21f-4583-9183-0be41f6613a9" containerName="extract"
Apr 22 20:03:53.266766 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.266749    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="78538159-e21f-4583-9183-0be41f6613a9" containerName="extract"
Apr 22 20:03:53.266971 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.266808    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="78538159-e21f-4583-9183-0be41f6613a9" containerName="extract"
Apr 22 20:03:53.272233 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.271199    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.274396 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.274370    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Apr 22 20:03:53.274543 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.274454    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Apr 22 20:03:53.274705 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.274688    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Apr 22 20:03:53.274768 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.274711    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-48vb9\""
Apr 22 20:03:53.274947 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.274933    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Apr 22 20:03:53.275062 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.275009    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"istio-ca-root-cert\""
Apr 22 20:03:53.275203 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.275182    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Apr 22 20:03:53.284523 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.284500    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"]
Apr 22 20:03:53.287983 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.287957    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6t4c\" (UniqueName: \"kubernetes.io/projected/3c2891c8-0663-48d2-8464-4db440e8d0d0-kube-api-access-z6t4c\") pod \"istiod-openshift-gateway-7cd77c7ffd-v8zwm\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.288070 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.288006    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3c2891c8-0663-48d2-8464-4db440e8d0d0-cacerts\") pod \"istiod-openshift-gateway-7cd77c7ffd-v8zwm\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.288070 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.288027    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7cd77c7ffd-v8zwm\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.288070 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.288044    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7cd77c7ffd-v8zwm\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.288070 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.288067    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2891c8-0663-48d2-8464-4db440e8d0d0-local-certs\") pod \"istiod-openshift-gateway-7cd77c7ffd-v8zwm\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.288236 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.288125    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7cd77c7ffd-v8zwm\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.288236 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.288227    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-token\") pod \"istiod-openshift-gateway-7cd77c7ffd-v8zwm\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.389566 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.389524    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z6t4c\" (UniqueName: \"kubernetes.io/projected/3c2891c8-0663-48d2-8464-4db440e8d0d0-kube-api-access-z6t4c\") pod \"istiod-openshift-gateway-7cd77c7ffd-v8zwm\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.389566 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.389563    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3c2891c8-0663-48d2-8464-4db440e8d0d0-cacerts\") pod \"istiod-openshift-gateway-7cd77c7ffd-v8zwm\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.389820 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.389582    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7cd77c7ffd-v8zwm\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.389820 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.389603    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7cd77c7ffd-v8zwm\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.389820 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.389733    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2891c8-0663-48d2-8464-4db440e8d0d0-local-certs\") pod \"istiod-openshift-gateway-7cd77c7ffd-v8zwm\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.389820 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.389792    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7cd77c7ffd-v8zwm\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.390025 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.389845    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-token\") pod \"istiod-openshift-gateway-7cd77c7ffd-v8zwm\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.390630 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.390604    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7cd77c7ffd-v8zwm\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.392141 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.392120    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2891c8-0663-48d2-8464-4db440e8d0d0-local-certs\") pod \"istiod-openshift-gateway-7cd77c7ffd-v8zwm\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.392141 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.392133    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7cd77c7ffd-v8zwm\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.392506 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.392487    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3c2891c8-0663-48d2-8464-4db440e8d0d0-cacerts\") pod \"istiod-openshift-gateway-7cd77c7ffd-v8zwm\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.392506 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.392497    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7cd77c7ffd-v8zwm\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.397074 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.397052    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-token\") pod \"istiod-openshift-gateway-7cd77c7ffd-v8zwm\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.397368 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.397348    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6t4c\" (UniqueName: \"kubernetes.io/projected/3c2891c8-0663-48d2-8464-4db440e8d0d0-kube-api-access-z6t4c\") pod \"istiod-openshift-gateway-7cd77c7ffd-v8zwm\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.581994 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.581899    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:53.727102 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.726945    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"]
Apr 22 20:03:53.730047 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:03:53.730002    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2891c8_0663_48d2_8464_4db440e8d0d0.slice/crio-5a42c94ebd431233a4d2ca73bf7082b561f1e79d83b8d2a4c470eac4acb4dc85 WatchSource:0}: Error finding container 5a42c94ebd431233a4d2ca73bf7082b561f1e79d83b8d2a4c470eac4acb4dc85: Status 404 returned error can't find the container with id 5a42c94ebd431233a4d2ca73bf7082b561f1e79d83b8d2a4c470eac4acb4dc85
Apr 22 20:03:53.838837 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:53.838744    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm" event={"ID":"3c2891c8-0663-48d2-8464-4db440e8d0d0","Type":"ContainerStarted","Data":"5a42c94ebd431233a4d2ca73bf7082b561f1e79d83b8d2a4c470eac4acb4dc85"}
Apr 22 20:03:57.343623 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:57.343583    2576 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Apr 22 20:03:57.344029 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:57.343654    2576 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Apr 22 20:03:57.855902 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:57.855865    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm" event={"ID":"3c2891c8-0663-48d2-8464-4db440e8d0d0","Type":"ContainerStarted","Data":"9925d66ce299e0b047bace232a5f373c9c91adc33a555678f4f136985631b62b"}
Apr 22 20:03:57.856107 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:57.855989    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:57.875961 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:57.875906    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm" podStartSLOduration=1.265086526 podStartE2EDuration="4.875888299s" podCreationTimestamp="2026-04-22 20:03:53 +0000 UTC" firstStartedPulling="2026-04-22 20:03:53.732536507 +0000 UTC m=+368.832492752" lastFinishedPulling="2026-04-22 20:03:57.343338279 +0000 UTC m=+372.443294525" observedRunningTime="2026-04-22 20:03:57.874355502 +0000 UTC m=+372.974311766" watchObservedRunningTime="2026-04-22 20:03:57.875888299 +0000 UTC m=+372.975844563"
Apr 22 20:03:58.861269 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:58.861232    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:03:59.933615 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:59.933582    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"]
Apr 22 20:03:59.937242 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:59.937217    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:03:59.940294 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:59.940264    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-pm9fd\""
Apr 22 20:03:59.952516 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:03:59.952488    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"]
Apr 22 20:04:00.052505 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.052472    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf701c3-b578-4cc8-a6af-4084c426e3ef-istio-token\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.052505 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.052513    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf701c3-b578-4cc8-a6af-4084c426e3ef-workload-certs\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.052733 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.052535    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf701c3-b578-4cc8-a6af-4084c426e3ef-istio-data\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.052733 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.052597    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf701c3-b578-4cc8-a6af-4084c426e3ef-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.052733 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.052630    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5pdp\" (UniqueName: \"kubernetes.io/projected/1cf701c3-b578-4cc8-a6af-4084c426e3ef-kube-api-access-x5pdp\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.052733 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.052680    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf701c3-b578-4cc8-a6af-4084c426e3ef-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.052859 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.052729    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf701c3-b578-4cc8-a6af-4084c426e3ef-workload-socket\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.052859 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.052789    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf701c3-b578-4cc8-a6af-4084c426e3ef-credential-socket\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.052859 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.052824    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf701c3-b578-4cc8-a6af-4084c426e3ef-istio-envoy\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.154253 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.154198    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf701c3-b578-4cc8-a6af-4084c426e3ef-istio-token\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.154253 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.154257    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf701c3-b578-4cc8-a6af-4084c426e3ef-workload-certs\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.154516 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.154277    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf701c3-b578-4cc8-a6af-4084c426e3ef-istio-data\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.154516 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.154295    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf701c3-b578-4cc8-a6af-4084c426e3ef-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.154516 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.154315    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x5pdp\" (UniqueName: \"kubernetes.io/projected/1cf701c3-b578-4cc8-a6af-4084c426e3ef-kube-api-access-x5pdp\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.154516 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.154356    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf701c3-b578-4cc8-a6af-4084c426e3ef-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.154516 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.154381    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf701c3-b578-4cc8-a6af-4084c426e3ef-workload-socket\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.154516 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.154432    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf701c3-b578-4cc8-a6af-4084c426e3ef-credential-socket\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.154516 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.154477    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf701c3-b578-4cc8-a6af-4084c426e3ef-istio-envoy\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.154862 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.154801    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf701c3-b578-4cc8-a6af-4084c426e3ef-istio-data\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.154862 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.154847    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf701c3-b578-4cc8-a6af-4084c426e3ef-workload-socket\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.154968 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.154911    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf701c3-b578-4cc8-a6af-4084c426e3ef-workload-certs\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.154968 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.154936    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf701c3-b578-4cc8-a6af-4084c426e3ef-credential-socket\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.155300 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.155283    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf701c3-b578-4cc8-a6af-4084c426e3ef-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.156860 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.156840    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf701c3-b578-4cc8-a6af-4084c426e3ef-istio-envoy\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.157252 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.157231    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf701c3-b578-4cc8-a6af-4084c426e3ef-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.162324 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.162293    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf701c3-b578-4cc8-a6af-4084c426e3ef-istio-token\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.162449 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.162415    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5pdp\" (UniqueName: \"kubernetes.io/projected/1cf701c3-b578-4cc8-a6af-4084c426e3ef-kube-api-access-x5pdp\") pod \"openshift-ai-inference-openshift-default-7c5447bb76-w9nb5\" (UID: \"1cf701c3-b578-4cc8-a6af-4084c426e3ef\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.251533 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.251442    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:00.383481 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.383454    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"]
Apr 22 20:04:00.385319 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:04:00.385288    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf701c3_b578_4cc8_a6af_4084c426e3ef.slice/crio-c4937ca7a9fba315819992e2143eaeb7ead31864fe434b244e0fa793a588eff8 WatchSource:0}: Error finding container c4937ca7a9fba315819992e2143eaeb7ead31864fe434b244e0fa793a588eff8: Status 404 returned error can't find the container with id c4937ca7a9fba315819992e2143eaeb7ead31864fe434b244e0fa793a588eff8
Apr 22 20:04:00.867413 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:00.867373    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5" event={"ID":"1cf701c3-b578-4cc8-a6af-4084c426e3ef","Type":"ContainerStarted","Data":"c4937ca7a9fba315819992e2143eaeb7ead31864fe434b244e0fa793a588eff8"}
Apr 22 20:04:03.489466 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:03.489407    2576 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Apr 22 20:04:03.489826 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:03.489506    2576 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Apr 22 20:04:03.489826 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:03.489544    2576 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Apr 22 20:04:03.841131 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:03.841040    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-operators/servicemesh-operator3-55f49c5f94-zr8xg"
Apr 22 20:04:03.884230 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:03.883411    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5" event={"ID":"1cf701c3-b578-4cc8-a6af-4084c426e3ef","Type":"ContainerStarted","Data":"a31f0ea166e31ea225db6ce764dc488fcf5f6630104c03c73d30f34fecbf6d22"}
Apr 22 20:04:03.907876 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:03.907812    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5" podStartSLOduration=1.806077464 podStartE2EDuration="4.907796427s" podCreationTimestamp="2026-04-22 20:03:59 +0000 UTC" firstStartedPulling="2026-04-22 20:04:00.387328147 +0000 UTC m=+375.487284391" lastFinishedPulling="2026-04-22 20:04:03.489047104 +0000 UTC m=+378.589003354" observedRunningTime="2026-04-22 20:04:03.905322703 +0000 UTC m=+379.005279024" watchObservedRunningTime="2026-04-22 20:04:03.907796427 +0000 UTC m=+379.007752693"
Apr 22 20:04:04.251986 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:04.251943    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:04.256671 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:04.256643    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:04.887082 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:04.887049    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:04.887953 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:04.887934    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-7c5447bb76-w9nb5"
Apr 22 20:04:07.640113 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.640073    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx"]
Apr 22 20:04:07.643715 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.643692    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx"
Apr 22 20:04:07.646268 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.646245    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Apr 22 20:04:07.646963 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.646948    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Apr 22 20:04:07.647033 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.646968    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4sls\""
Apr 22 20:04:07.652440 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.652412    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx"]
Apr 22 20:04:07.721790 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.721741    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwxqt\" (UniqueName: \"kubernetes.io/projected/90b3850b-2feb-41e8-8eb8-123f5021792f-kube-api-access-vwxqt\") pod \"5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx\" (UID: \"90b3850b-2feb-41e8-8eb8-123f5021792f\") " pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx"
Apr 22 20:04:07.721790 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.721794    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90b3850b-2feb-41e8-8eb8-123f5021792f-bundle\") pod \"5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx\" (UID: \"90b3850b-2feb-41e8-8eb8-123f5021792f\") " pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx"
Apr 22 20:04:07.722012 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.721812    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90b3850b-2feb-41e8-8eb8-123f5021792f-util\") pod \"5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx\" (UID: \"90b3850b-2feb-41e8-8eb8-123f5021792f\") " pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx"
Apr 22 20:04:07.739964 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.739928    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q"]
Apr 22 20:04:07.743599 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.743581    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q"
Apr 22 20:04:07.750864 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.750834    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q"]
Apr 22 20:04:07.822624 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.822587    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vwxqt\" (UniqueName: \"kubernetes.io/projected/90b3850b-2feb-41e8-8eb8-123f5021792f-kube-api-access-vwxqt\") pod \"5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx\" (UID: \"90b3850b-2feb-41e8-8eb8-123f5021792f\") " pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx"
Apr 22 20:04:07.822836 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.822641    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90b3850b-2feb-41e8-8eb8-123f5021792f-bundle\") pod \"5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx\" (UID: \"90b3850b-2feb-41e8-8eb8-123f5021792f\") " pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx"
Apr 22 20:04:07.822836 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.822813    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90b3850b-2feb-41e8-8eb8-123f5021792f-util\") pod \"5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx\" (UID: \"90b3850b-2feb-41e8-8eb8-123f5021792f\") " pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx"
Apr 22 20:04:07.822957 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.822887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cce66f5f-4df1-4f70-bcc2-fc4105d458a7-util\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q\" (UID: \"cce66f5f-4df1-4f70-bcc2-fc4105d458a7\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q"
Apr 22 20:04:07.822957 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.822921    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cce66f5f-4df1-4f70-bcc2-fc4105d458a7-bundle\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q\" (UID: \"cce66f5f-4df1-4f70-bcc2-fc4105d458a7\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q"
Apr 22 20:04:07.823066 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.822989    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gchpn\" (UniqueName: \"kubernetes.io/projected/cce66f5f-4df1-4f70-bcc2-fc4105d458a7-kube-api-access-gchpn\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q\" (UID: \"cce66f5f-4df1-4f70-bcc2-fc4105d458a7\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q"
Apr 22 20:04:07.823066 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.823033    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90b3850b-2feb-41e8-8eb8-123f5021792f-bundle\") pod \"5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx\" (UID: \"90b3850b-2feb-41e8-8eb8-123f5021792f\") " pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx"
Apr 22 20:04:07.823238 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.823220    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90b3850b-2feb-41e8-8eb8-123f5021792f-util\") pod \"5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx\" (UID: \"90b3850b-2feb-41e8-8eb8-123f5021792f\") " pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx"
Apr 22 20:04:07.832222 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.832083    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwxqt\" (UniqueName: \"kubernetes.io/projected/90b3850b-2feb-41e8-8eb8-123f5021792f-kube-api-access-vwxqt\") pod \"5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx\" (UID: \"90b3850b-2feb-41e8-8eb8-123f5021792f\") " pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx"
Apr 22 20:04:07.841315 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.841289    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg"]
Apr 22 20:04:07.845031 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.845015    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg"
Apr 22 20:04:07.852206 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.852181    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg"]
Apr 22 20:04:07.923834 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.923747    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cce66f5f-4df1-4f70-bcc2-fc4105d458a7-util\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q\" (UID: \"cce66f5f-4df1-4f70-bcc2-fc4105d458a7\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q"
Apr 22 20:04:07.923834 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.923789    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/195ccb7d-b072-4c8b-b4a0-efceb181bff3-util\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg\" (UID: \"195ccb7d-b072-4c8b-b4a0-efceb181bff3\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg"
Apr 22 20:04:07.923834 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.923815    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cce66f5f-4df1-4f70-bcc2-fc4105d458a7-bundle\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q\" (UID: \"cce66f5f-4df1-4f70-bcc2-fc4105d458a7\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q"
Apr 22 20:04:07.924042 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.923835    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/195ccb7d-b072-4c8b-b4a0-efceb181bff3-bundle\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg\" (UID: \"195ccb7d-b072-4c8b-b4a0-efceb181bff3\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg"
Apr 22 20:04:07.924042 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.923907    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88rkm\" (UniqueName: \"kubernetes.io/projected/195ccb7d-b072-4c8b-b4a0-efceb181bff3-kube-api-access-88rkm\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg\" (UID: \"195ccb7d-b072-4c8b-b4a0-efceb181bff3\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg"
Apr 22 20:04:07.924042 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.923952    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gchpn\" (UniqueName: \"kubernetes.io/projected/cce66f5f-4df1-4f70-bcc2-fc4105d458a7-kube-api-access-gchpn\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q\" (UID: \"cce66f5f-4df1-4f70-bcc2-fc4105d458a7\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q"
Apr 22 20:04:07.924230 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.924209    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cce66f5f-4df1-4f70-bcc2-fc4105d458a7-util\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q\" (UID: \"cce66f5f-4df1-4f70-bcc2-fc4105d458a7\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q"
Apr 22 20:04:07.924279 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.924220    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cce66f5f-4df1-4f70-bcc2-fc4105d458a7-bundle\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q\" (UID: \"cce66f5f-4df1-4f70-bcc2-fc4105d458a7\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q"
Apr 22 20:04:07.933693 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.933667    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gchpn\" (UniqueName: \"kubernetes.io/projected/cce66f5f-4df1-4f70-bcc2-fc4105d458a7-kube-api-access-gchpn\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q\" (UID: \"cce66f5f-4df1-4f70-bcc2-fc4105d458a7\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q"
Apr 22 20:04:07.945745 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.945720    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr"]
Apr 22 20:04:07.949307 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.949293    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr"
Apr 22 20:04:07.954631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.954606    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx"
Apr 22 20:04:07.958785 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:07.958760    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr"]
Apr 22 20:04:08.025090 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.025061    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/195ccb7d-b072-4c8b-b4a0-efceb181bff3-util\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg\" (UID: \"195ccb7d-b072-4c8b-b4a0-efceb181bff3\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg"
Apr 22 20:04:08.025276 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.025103    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13d2cd14-2cb4-4a8a-bd44-8746281387a7-util\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr\" (UID: \"13d2cd14-2cb4-4a8a-bd44-8746281387a7\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr"
Apr 22 20:04:08.025276 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.025130    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/195ccb7d-b072-4c8b-b4a0-efceb181bff3-bundle\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg\" (UID: \"195ccb7d-b072-4c8b-b4a0-efceb181bff3\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg"
Apr 22 20:04:08.025387 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.025285    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-88rkm\" (UniqueName: \"kubernetes.io/projected/195ccb7d-b072-4c8b-b4a0-efceb181bff3-kube-api-access-88rkm\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg\" (UID: \"195ccb7d-b072-4c8b-b4a0-efceb181bff3\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg"
Apr 22 20:04:08.025387 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.025346    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdvld\" (UniqueName: \"kubernetes.io/projected/13d2cd14-2cb4-4a8a-bd44-8746281387a7-kube-api-access-jdvld\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr\" (UID: \"13d2cd14-2cb4-4a8a-bd44-8746281387a7\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr"
Apr 22 20:04:08.025508 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.025384    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13d2cd14-2cb4-4a8a-bd44-8746281387a7-bundle\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr\" (UID: \"13d2cd14-2cb4-4a8a-bd44-8746281387a7\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr"
Apr 22 20:04:08.025572 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.025552    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/195ccb7d-b072-4c8b-b4a0-efceb181bff3-util\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg\" (UID: \"195ccb7d-b072-4c8b-b4a0-efceb181bff3\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg"
Apr 22 20:04:08.025616 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.025583    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/195ccb7d-b072-4c8b-b4a0-efceb181bff3-bundle\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg\" (UID: \"195ccb7d-b072-4c8b-b4a0-efceb181bff3\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg"
Apr 22 20:04:08.034355 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.034331    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88rkm\" (UniqueName: \"kubernetes.io/projected/195ccb7d-b072-4c8b-b4a0-efceb181bff3-kube-api-access-88rkm\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg\" (UID: \"195ccb7d-b072-4c8b-b4a0-efceb181bff3\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg"
Apr 22 20:04:08.053498 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.053467    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q"
Apr 22 20:04:08.090399 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.090366    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx"]
Apr 22 20:04:08.092244 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:04:08.092205    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b3850b_2feb_41e8_8eb8_123f5021792f.slice/crio-ff25464cf38144ece5cc6807d1eea695f45e8da90411251d91acd362d2959206 WatchSource:0}: Error finding container ff25464cf38144ece5cc6807d1eea695f45e8da90411251d91acd362d2959206: Status 404 returned error can't find the container with id ff25464cf38144ece5cc6807d1eea695f45e8da90411251d91acd362d2959206
Apr 22 20:04:08.126883 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.126790    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13d2cd14-2cb4-4a8a-bd44-8746281387a7-util\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr\" (UID: \"13d2cd14-2cb4-4a8a-bd44-8746281387a7\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr"
Apr 22 20:04:08.126883 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.126849    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvld\" (UniqueName: \"kubernetes.io/projected/13d2cd14-2cb4-4a8a-bd44-8746281387a7-kube-api-access-jdvld\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr\" (UID: \"13d2cd14-2cb4-4a8a-bd44-8746281387a7\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr"
Apr 22 20:04:08.126883 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.126877    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13d2cd14-2cb4-4a8a-bd44-8746281387a7-bundle\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr\" (UID: \"13d2cd14-2cb4-4a8a-bd44-8746281387a7\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr"
Apr 22 20:04:08.127178 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.127136    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13d2cd14-2cb4-4a8a-bd44-8746281387a7-util\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr\" (UID: \"13d2cd14-2cb4-4a8a-bd44-8746281387a7\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr"
Apr 22 20:04:08.127303 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.127285    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13d2cd14-2cb4-4a8a-bd44-8746281387a7-bundle\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr\" (UID: \"13d2cd14-2cb4-4a8a-bd44-8746281387a7\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr"
Apr 22 20:04:08.136509 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.136480    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvld\" (UniqueName: \"kubernetes.io/projected/13d2cd14-2cb4-4a8a-bd44-8746281387a7-kube-api-access-jdvld\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr\" (UID: \"13d2cd14-2cb4-4a8a-bd44-8746281387a7\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr"
Apr 22 20:04:08.156197 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.156167    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg"
Apr 22 20:04:08.195624 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.195594    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q"]
Apr 22 20:04:08.196813 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:04:08.196789    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce66f5f_4df1_4f70_bcc2_fc4105d458a7.slice/crio-6dc6f0de6332cb57c22adfc51ae0cc8a963d3220f44c44fac53b320305232f3c WatchSource:0}: Error finding container 6dc6f0de6332cb57c22adfc51ae0cc8a963d3220f44c44fac53b320305232f3c: Status 404 returned error can't find the container with id 6dc6f0de6332cb57c22adfc51ae0cc8a963d3220f44c44fac53b320305232f3c
Apr 22 20:04:08.259693 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.259665    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr"
Apr 22 20:04:08.295085 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.295059    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg"]
Apr 22 20:04:08.298167 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:04:08.298116    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195ccb7d_b072_4c8b_b4a0_efceb181bff3.slice/crio-befe4e593c7fdef3f09261a39394d19c041a977542aa7668c84b3df44d451cff WatchSource:0}: Error finding container befe4e593c7fdef3f09261a39394d19c041a977542aa7668c84b3df44d451cff: Status 404 returned error can't find the container with id befe4e593c7fdef3f09261a39394d19c041a977542aa7668c84b3df44d451cff
Apr 22 20:04:08.398847 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.398819    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr"]
Apr 22 20:04:08.401429 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:04:08.401400    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d2cd14_2cb4_4a8a_bd44_8746281387a7.slice/crio-e472234453e6c96a6390cb8198ebe8d54efb0cf88dc8df68854c606e22b30a13 WatchSource:0}: Error finding container e472234453e6c96a6390cb8198ebe8d54efb0cf88dc8df68854c606e22b30a13: Status 404 returned error can't find the container with id e472234453e6c96a6390cb8198ebe8d54efb0cf88dc8df68854c606e22b30a13
Apr 22 20:04:08.902286 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.902254    2576 generic.go:358] "Generic (PLEG): container finished" podID="195ccb7d-b072-4c8b-b4a0-efceb181bff3" containerID="2bdb1190372a87b68f5fc57f41f64c83c25cd124cc6f7bcff2055ab8aaac4dc0" exitCode=0
Apr 22 20:04:08.902774 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.902323    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg" event={"ID":"195ccb7d-b072-4c8b-b4a0-efceb181bff3","Type":"ContainerDied","Data":"2bdb1190372a87b68f5fc57f41f64c83c25cd124cc6f7bcff2055ab8aaac4dc0"}
Apr 22 20:04:08.902774 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.902350    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg" event={"ID":"195ccb7d-b072-4c8b-b4a0-efceb181bff3","Type":"ContainerStarted","Data":"befe4e593c7fdef3f09261a39394d19c041a977542aa7668c84b3df44d451cff"}
Apr 22 20:04:08.903773 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.903735    2576 generic.go:358] "Generic (PLEG): container finished" podID="90b3850b-2feb-41e8-8eb8-123f5021792f" containerID="5e87eed87768f43675b290982d157501d9304b73345d2cc1904507cc28a37593" exitCode=0
Apr 22 20:04:08.903882 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.903785    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx" event={"ID":"90b3850b-2feb-41e8-8eb8-123f5021792f","Type":"ContainerDied","Data":"5e87eed87768f43675b290982d157501d9304b73345d2cc1904507cc28a37593"}
Apr 22 20:04:08.903882 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.903813    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx" event={"ID":"90b3850b-2feb-41e8-8eb8-123f5021792f","Type":"ContainerStarted","Data":"ff25464cf38144ece5cc6807d1eea695f45e8da90411251d91acd362d2959206"}
Apr 22 20:04:08.905272 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.905230    2576 generic.go:358] "Generic (PLEG): container finished" podID="13d2cd14-2cb4-4a8a-bd44-8746281387a7" containerID="8f7d19dbd29b6d6fb4bac186f9650e37af5e0e2ba19709c89a3b9666e2f6fd31" exitCode=0
Apr 22 20:04:08.905366 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.905268    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr" event={"ID":"13d2cd14-2cb4-4a8a-bd44-8746281387a7","Type":"ContainerDied","Data":"8f7d19dbd29b6d6fb4bac186f9650e37af5e0e2ba19709c89a3b9666e2f6fd31"}
Apr 22 20:04:08.905366 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.905295    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr" event={"ID":"13d2cd14-2cb4-4a8a-bd44-8746281387a7","Type":"ContainerStarted","Data":"e472234453e6c96a6390cb8198ebe8d54efb0cf88dc8df68854c606e22b30a13"}
Apr 22 20:04:08.906831 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.906807    2576 generic.go:358] "Generic (PLEG): container finished" podID="cce66f5f-4df1-4f70-bcc2-fc4105d458a7" containerID="582ef4f372279c86b1edf4cb678e7fc69248eb87691b6f9a1b6d0490e1358278" exitCode=0
Apr 22 20:04:08.906941 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.906861    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q" event={"ID":"cce66f5f-4df1-4f70-bcc2-fc4105d458a7","Type":"ContainerDied","Data":"582ef4f372279c86b1edf4cb678e7fc69248eb87691b6f9a1b6d0490e1358278"}
Apr 22 20:04:08.906941 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:08.906882    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q" event={"ID":"cce66f5f-4df1-4f70-bcc2-fc4105d458a7","Type":"ContainerStarted","Data":"6dc6f0de6332cb57c22adfc51ae0cc8a963d3220f44c44fac53b320305232f3c"}
Apr 22 20:04:09.912351 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:09.912318    2576 generic.go:358] "Generic (PLEG): container finished" podID="195ccb7d-b072-4c8b-b4a0-efceb181bff3" containerID="cdbac995342587f29a3f9b0db19ad28bfc468033ad819394aa5205e62bd35aa9" exitCode=0
Apr 22 20:04:09.912749 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:09.912358    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg" event={"ID":"195ccb7d-b072-4c8b-b4a0-efceb181bff3","Type":"ContainerDied","Data":"cdbac995342587f29a3f9b0db19ad28bfc468033ad819394aa5205e62bd35aa9"}
Apr 22 20:04:09.914202 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:09.914176    2576 generic.go:358] "Generic (PLEG): container finished" podID="13d2cd14-2cb4-4a8a-bd44-8746281387a7" containerID="452533eda96ea1d2c5e833f22d93ea74c8f42f8bdf1fbb7b3a6e535c7d673ecc" exitCode=0
Apr 22 20:04:09.914274 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:09.914259    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr" event={"ID":"13d2cd14-2cb4-4a8a-bd44-8746281387a7","Type":"ContainerDied","Data":"452533eda96ea1d2c5e833f22d93ea74c8f42f8bdf1fbb7b3a6e535c7d673ecc"}
Apr 22 20:04:10.920344 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:10.920308    2576 generic.go:358] "Generic (PLEG): container finished" podID="13d2cd14-2cb4-4a8a-bd44-8746281387a7" containerID="448ccdf40eaeaea4620208ce2eacc02c1dedeaf4a760dd9acdc18e94c1549a51" exitCode=0
Apr 22 20:04:10.920704 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:10.920389    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr" event={"ID":"13d2cd14-2cb4-4a8a-bd44-8746281387a7","Type":"ContainerDied","Data":"448ccdf40eaeaea4620208ce2eacc02c1dedeaf4a760dd9acdc18e94c1549a51"}
Apr 22 20:04:10.922259 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:10.922234    2576 generic.go:358] "Generic (PLEG): container finished" podID="195ccb7d-b072-4c8b-b4a0-efceb181bff3" containerID="d359a7836fe45193ac56bbd0d4d57182c44451fa126aabea8d92b6371be2c4b7" exitCode=0
Apr 22 20:04:10.922386 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:10.922295    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg" event={"ID":"195ccb7d-b072-4c8b-b4a0-efceb181bff3","Type":"ContainerDied","Data":"d359a7836fe45193ac56bbd0d4d57182c44451fa126aabea8d92b6371be2c4b7"}
Apr 22 20:04:10.923813 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:10.923785    2576 generic.go:358] "Generic (PLEG): container finished" podID="90b3850b-2feb-41e8-8eb8-123f5021792f" containerID="e7364aca9e1fbd370641f936e658bd3b21354bc579b42db7bf52294171eefdfe" exitCode=0
Apr 22 20:04:10.923900 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:10.923846    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx" event={"ID":"90b3850b-2feb-41e8-8eb8-123f5021792f","Type":"ContainerDied","Data":"e7364aca9e1fbd370641f936e658bd3b21354bc579b42db7bf52294171eefdfe"}
Apr 22 20:04:11.929360 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:11.929324    2576 generic.go:358] "Generic (PLEG): container finished" podID="90b3850b-2feb-41e8-8eb8-123f5021792f" containerID="f7e75ad058cd95da05f25ecc4088437e8ff26591fa8049ae4523b6a3cbd6fa81" exitCode=0
Apr 22 20:04:11.929741 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:11.929413    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx" event={"ID":"90b3850b-2feb-41e8-8eb8-123f5021792f","Type":"ContainerDied","Data":"f7e75ad058cd95da05f25ecc4088437e8ff26591fa8049ae4523b6a3cbd6fa81"}
Apr 22 20:04:12.078772 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.078751    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg"
Apr 22 20:04:12.101509 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.101478    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr"
Apr 22 20:04:12.166566 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.166531    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdvld\" (UniqueName: \"kubernetes.io/projected/13d2cd14-2cb4-4a8a-bd44-8746281387a7-kube-api-access-jdvld\") pod \"13d2cd14-2cb4-4a8a-bd44-8746281387a7\" (UID: \"13d2cd14-2cb4-4a8a-bd44-8746281387a7\") "
Apr 22 20:04:12.166720 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.166577    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13d2cd14-2cb4-4a8a-bd44-8746281387a7-bundle\") pod \"13d2cd14-2cb4-4a8a-bd44-8746281387a7\" (UID: \"13d2cd14-2cb4-4a8a-bd44-8746281387a7\") "
Apr 22 20:04:12.166720 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.166602    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/195ccb7d-b072-4c8b-b4a0-efceb181bff3-util\") pod \"195ccb7d-b072-4c8b-b4a0-efceb181bff3\" (UID: \"195ccb7d-b072-4c8b-b4a0-efceb181bff3\") "
Apr 22 20:04:12.166720 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.166625    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13d2cd14-2cb4-4a8a-bd44-8746281387a7-util\") pod \"13d2cd14-2cb4-4a8a-bd44-8746281387a7\" (UID: \"13d2cd14-2cb4-4a8a-bd44-8746281387a7\") "
Apr 22 20:04:12.166720 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.166654    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/195ccb7d-b072-4c8b-b4a0-efceb181bff3-bundle\") pod \"195ccb7d-b072-4c8b-b4a0-efceb181bff3\" (UID: \"195ccb7d-b072-4c8b-b4a0-efceb181bff3\") "
Apr 22 20:04:12.166720 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.166718    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88rkm\" (UniqueName: \"kubernetes.io/projected/195ccb7d-b072-4c8b-b4a0-efceb181bff3-kube-api-access-88rkm\") pod \"195ccb7d-b072-4c8b-b4a0-efceb181bff3\" (UID: \"195ccb7d-b072-4c8b-b4a0-efceb181bff3\") "
Apr 22 20:04:12.167349 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.167319    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195ccb7d-b072-4c8b-b4a0-efceb181bff3-bundle" (OuterVolumeSpecName: "bundle") pod "195ccb7d-b072-4c8b-b4a0-efceb181bff3" (UID: "195ccb7d-b072-4c8b-b4a0-efceb181bff3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:04:12.167445 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.167342    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d2cd14-2cb4-4a8a-bd44-8746281387a7-bundle" (OuterVolumeSpecName: "bundle") pod "13d2cd14-2cb4-4a8a-bd44-8746281387a7" (UID: "13d2cd14-2cb4-4a8a-bd44-8746281387a7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:04:12.169124 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.169095    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195ccb7d-b072-4c8b-b4a0-efceb181bff3-kube-api-access-88rkm" (OuterVolumeSpecName: "kube-api-access-88rkm") pod "195ccb7d-b072-4c8b-b4a0-efceb181bff3" (UID: "195ccb7d-b072-4c8b-b4a0-efceb181bff3"). InnerVolumeSpecName "kube-api-access-88rkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:04:12.169243 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.169201    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d2cd14-2cb4-4a8a-bd44-8746281387a7-kube-api-access-jdvld" (OuterVolumeSpecName: "kube-api-access-jdvld") pod "13d2cd14-2cb4-4a8a-bd44-8746281387a7" (UID: "13d2cd14-2cb4-4a8a-bd44-8746281387a7"). InnerVolumeSpecName "kube-api-access-jdvld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:04:12.172392 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.172359    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d2cd14-2cb4-4a8a-bd44-8746281387a7-util" (OuterVolumeSpecName: "util") pod "13d2cd14-2cb4-4a8a-bd44-8746281387a7" (UID: "13d2cd14-2cb4-4a8a-bd44-8746281387a7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:04:12.175030 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.175006    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195ccb7d-b072-4c8b-b4a0-efceb181bff3-util" (OuterVolumeSpecName: "util") pod "195ccb7d-b072-4c8b-b4a0-efceb181bff3" (UID: "195ccb7d-b072-4c8b-b4a0-efceb181bff3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:04:12.267762 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.267677    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdvld\" (UniqueName: \"kubernetes.io/projected/13d2cd14-2cb4-4a8a-bd44-8746281387a7-kube-api-access-jdvld\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:04:12.267762 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.267710    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13d2cd14-2cb4-4a8a-bd44-8746281387a7-bundle\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:04:12.267762 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.267720    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/195ccb7d-b072-4c8b-b4a0-efceb181bff3-util\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:04:12.267762 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.267728    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13d2cd14-2cb4-4a8a-bd44-8746281387a7-util\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:04:12.267762 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.267736    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/195ccb7d-b072-4c8b-b4a0-efceb181bff3-bundle\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:04:12.267762 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.267744    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88rkm\" (UniqueName: \"kubernetes.io/projected/195ccb7d-b072-4c8b-b4a0-efceb181bff3-kube-api-access-88rkm\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:04:12.935046 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.935013    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr"
Apr 22 20:04:12.935501 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.935017    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bps5rr" event={"ID":"13d2cd14-2cb4-4a8a-bd44-8746281387a7","Type":"ContainerDied","Data":"e472234453e6c96a6390cb8198ebe8d54efb0cf88dc8df68854c606e22b30a13"}
Apr 22 20:04:12.935501 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.935118    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e472234453e6c96a6390cb8198ebe8d54efb0cf88dc8df68854c606e22b30a13"
Apr 22 20:04:12.936788 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.936768    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg"
Apr 22 20:04:12.936788 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.936778    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885mkzg" event={"ID":"195ccb7d-b072-4c8b-b4a0-efceb181bff3","Type":"ContainerDied","Data":"befe4e593c7fdef3f09261a39394d19c041a977542aa7668c84b3df44d451cff"}
Apr 22 20:04:12.936965 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:12.936803    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="befe4e593c7fdef3f09261a39394d19c041a977542aa7668c84b3df44d451cff"
Apr 22 20:04:13.048956 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:13.048934    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx"
Apr 22 20:04:13.176926 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:13.176896    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90b3850b-2feb-41e8-8eb8-123f5021792f-util\") pod \"90b3850b-2feb-41e8-8eb8-123f5021792f\" (UID: \"90b3850b-2feb-41e8-8eb8-123f5021792f\") "
Apr 22 20:04:13.177063 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:13.177005    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwxqt\" (UniqueName: \"kubernetes.io/projected/90b3850b-2feb-41e8-8eb8-123f5021792f-kube-api-access-vwxqt\") pod \"90b3850b-2feb-41e8-8eb8-123f5021792f\" (UID: \"90b3850b-2feb-41e8-8eb8-123f5021792f\") "
Apr 22 20:04:13.177063 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:13.177045    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90b3850b-2feb-41e8-8eb8-123f5021792f-bundle\") pod \"90b3850b-2feb-41e8-8eb8-123f5021792f\" (UID: \"90b3850b-2feb-41e8-8eb8-123f5021792f\") "
Apr 22 20:04:13.177941 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:13.177848    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b3850b-2feb-41e8-8eb8-123f5021792f-bundle" (OuterVolumeSpecName: "bundle") pod "90b3850b-2feb-41e8-8eb8-123f5021792f" (UID: "90b3850b-2feb-41e8-8eb8-123f5021792f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:04:13.179526 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:13.179473    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b3850b-2feb-41e8-8eb8-123f5021792f-kube-api-access-vwxqt" (OuterVolumeSpecName: "kube-api-access-vwxqt") pod "90b3850b-2feb-41e8-8eb8-123f5021792f" (UID: "90b3850b-2feb-41e8-8eb8-123f5021792f"). InnerVolumeSpecName "kube-api-access-vwxqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:04:13.182382 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:13.182360    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b3850b-2feb-41e8-8eb8-123f5021792f-util" (OuterVolumeSpecName: "util") pod "90b3850b-2feb-41e8-8eb8-123f5021792f" (UID: "90b3850b-2feb-41e8-8eb8-123f5021792f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:04:13.277795 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:13.277758    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90b3850b-2feb-41e8-8eb8-123f5021792f-bundle\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:04:13.277795 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:13.277792    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90b3850b-2feb-41e8-8eb8-123f5021792f-util\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:04:13.277795 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:13.277802    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwxqt\" (UniqueName: \"kubernetes.io/projected/90b3850b-2feb-41e8-8eb8-123f5021792f-kube-api-access-vwxqt\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:04:13.941917 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:13.941879    2576 generic.go:358] "Generic (PLEG): container finished" podID="cce66f5f-4df1-4f70-bcc2-fc4105d458a7" containerID="c7119db22705349e6d37abb2bed98fd3b53034c36e615b9055c29bbf52bece6c" exitCode=0
Apr 22 20:04:13.942441 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:13.941983    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q" event={"ID":"cce66f5f-4df1-4f70-bcc2-fc4105d458a7","Type":"ContainerDied","Data":"c7119db22705349e6d37abb2bed98fd3b53034c36e615b9055c29bbf52bece6c"}
Apr 22 20:04:13.943732 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:13.943660    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx" event={"ID":"90b3850b-2feb-41e8-8eb8-123f5021792f","Type":"ContainerDied","Data":"ff25464cf38144ece5cc6807d1eea695f45e8da90411251d91acd362d2959206"}
Apr 22 20:04:13.943732 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:13.943684    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff25464cf38144ece5cc6807d1eea695f45e8da90411251d91acd362d2959206"
Apr 22 20:04:13.943829 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:13.943748    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e503dlbwx"
Apr 22 20:04:14.948737 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:14.948706    2576 generic.go:358] "Generic (PLEG): container finished" podID="cce66f5f-4df1-4f70-bcc2-fc4105d458a7" containerID="8875fa43a17449a6a511a1d4a514e2f3b618f875d069d5d19bf8732b0641688b" exitCode=0
Apr 22 20:04:14.949124 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:14.948796    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q" event={"ID":"cce66f5f-4df1-4f70-bcc2-fc4105d458a7","Type":"ContainerDied","Data":"8875fa43a17449a6a511a1d4a514e2f3b618f875d069d5d19bf8732b0641688b"}
Apr 22 20:04:16.077682 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:16.077661    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q"
Apr 22 20:04:16.202049 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:16.202015    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gchpn\" (UniqueName: \"kubernetes.io/projected/cce66f5f-4df1-4f70-bcc2-fc4105d458a7-kube-api-access-gchpn\") pod \"cce66f5f-4df1-4f70-bcc2-fc4105d458a7\" (UID: \"cce66f5f-4df1-4f70-bcc2-fc4105d458a7\") "
Apr 22 20:04:16.202246 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:16.202076    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cce66f5f-4df1-4f70-bcc2-fc4105d458a7-bundle\") pod \"cce66f5f-4df1-4f70-bcc2-fc4105d458a7\" (UID: \"cce66f5f-4df1-4f70-bcc2-fc4105d458a7\") "
Apr 22 20:04:16.202246 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:16.202094    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cce66f5f-4df1-4f70-bcc2-fc4105d458a7-util\") pod \"cce66f5f-4df1-4f70-bcc2-fc4105d458a7\" (UID: \"cce66f5f-4df1-4f70-bcc2-fc4105d458a7\") "
Apr 22 20:04:16.202761 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:16.202710    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce66f5f-4df1-4f70-bcc2-fc4105d458a7-bundle" (OuterVolumeSpecName: "bundle") pod "cce66f5f-4df1-4f70-bcc2-fc4105d458a7" (UID: "cce66f5f-4df1-4f70-bcc2-fc4105d458a7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:04:16.204215 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:16.204162    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce66f5f-4df1-4f70-bcc2-fc4105d458a7-kube-api-access-gchpn" (OuterVolumeSpecName: "kube-api-access-gchpn") pod "cce66f5f-4df1-4f70-bcc2-fc4105d458a7" (UID: "cce66f5f-4df1-4f70-bcc2-fc4105d458a7"). InnerVolumeSpecName "kube-api-access-gchpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:04:16.206471 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:16.206451    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce66f5f-4df1-4f70-bcc2-fc4105d458a7-util" (OuterVolumeSpecName: "util") pod "cce66f5f-4df1-4f70-bcc2-fc4105d458a7" (UID: "cce66f5f-4df1-4f70-bcc2-fc4105d458a7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:04:16.303733 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:16.303680    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gchpn\" (UniqueName: \"kubernetes.io/projected/cce66f5f-4df1-4f70-bcc2-fc4105d458a7-kube-api-access-gchpn\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:04:16.303733 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:16.303726    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cce66f5f-4df1-4f70-bcc2-fc4105d458a7-bundle\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:04:16.303733 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:16.303737    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cce66f5f-4df1-4f70-bcc2-fc4105d458a7-util\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:04:16.958317 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:16.958284    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q"
Apr 22 20:04:16.958479 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:16.958288    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c308wp5q" event={"ID":"cce66f5f-4df1-4f70-bcc2-fc4105d458a7","Type":"ContainerDied","Data":"6dc6f0de6332cb57c22adfc51ae0cc8a963d3220f44c44fac53b320305232f3c"}
Apr 22 20:04:16.958479 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:16.958394    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dc6f0de6332cb57c22adfc51ae0cc8a963d3220f44c44fac53b320305232f3c"
Apr 22 20:04:21.381005 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.380971    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bd97b97cc-rnh5p"]
Apr 22 20:04:21.381376 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381318    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="195ccb7d-b072-4c8b-b4a0-efceb181bff3" containerName="pull"
Apr 22 20:04:21.381376 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381329    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="195ccb7d-b072-4c8b-b4a0-efceb181bff3" containerName="pull"
Apr 22 20:04:21.381376 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381341    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d2cd14-2cb4-4a8a-bd44-8746281387a7" containerName="util"
Apr 22 20:04:21.381376 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381346    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d2cd14-2cb4-4a8a-bd44-8746281387a7" containerName="util"
Apr 22 20:04:21.381376 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381353    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce66f5f-4df1-4f70-bcc2-fc4105d458a7" containerName="extract"
Apr 22 20:04:21.381376 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381359    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce66f5f-4df1-4f70-bcc2-fc4105d458a7" containerName="extract"
Apr 22 20:04:21.381376 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381368    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d2cd14-2cb4-4a8a-bd44-8746281387a7" containerName="extract"
Apr 22 20:04:21.381376 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381374    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d2cd14-2cb4-4a8a-bd44-8746281387a7" containerName="extract"
Apr 22 20:04:21.381631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381383    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90b3850b-2feb-41e8-8eb8-123f5021792f" containerName="extract"
Apr 22 20:04:21.381631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381388    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="90b3850b-2feb-41e8-8eb8-123f5021792f" containerName="extract"
Apr 22 20:04:21.381631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381394    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d2cd14-2cb4-4a8a-bd44-8746281387a7" containerName="pull"
Apr 22 20:04:21.381631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381399    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d2cd14-2cb4-4a8a-bd44-8746281387a7" containerName="pull"
Apr 22 20:04:21.381631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381408    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce66f5f-4df1-4f70-bcc2-fc4105d458a7" containerName="util"
Apr 22 20:04:21.381631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381412    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce66f5f-4df1-4f70-bcc2-fc4105d458a7" containerName="util"
Apr 22 20:04:21.381631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381417    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="195ccb7d-b072-4c8b-b4a0-efceb181bff3" containerName="util"
Apr 22 20:04:21.381631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381421    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="195ccb7d-b072-4c8b-b4a0-efceb181bff3" containerName="util"
Apr 22 20:04:21.381631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381426    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce66f5f-4df1-4f70-bcc2-fc4105d458a7" containerName="pull"
Apr 22 20:04:21.381631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381431    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce66f5f-4df1-4f70-bcc2-fc4105d458a7" containerName="pull"
Apr 22 20:04:21.381631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381438    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90b3850b-2feb-41e8-8eb8-123f5021792f" containerName="pull"
Apr 22 20:04:21.381631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381443    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="90b3850b-2feb-41e8-8eb8-123f5021792f" containerName="pull"
Apr 22 20:04:21.381631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381448    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="195ccb7d-b072-4c8b-b4a0-efceb181bff3" containerName="extract"
Apr 22 20:04:21.381631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381453    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="195ccb7d-b072-4c8b-b4a0-efceb181bff3" containerName="extract"
Apr 22 20:04:21.381631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381470    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90b3850b-2feb-41e8-8eb8-123f5021792f" containerName="util"
Apr 22 20:04:21.381631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381475    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="90b3850b-2feb-41e8-8eb8-123f5021792f" containerName="util"
Apr 22 20:04:21.381631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381521    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="cce66f5f-4df1-4f70-bcc2-fc4105d458a7" containerName="extract"
Apr 22 20:04:21.381631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381529    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="13d2cd14-2cb4-4a8a-bd44-8746281387a7" containerName="extract"
Apr 22 20:04:21.381631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381537    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="90b3850b-2feb-41e8-8eb8-123f5021792f" containerName="extract"
Apr 22 20:04:21.381631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.381545    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="195ccb7d-b072-4c8b-b4a0-efceb181bff3" containerName="extract"
Apr 22 20:04:21.386308 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.386283    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.398458 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.398432    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bd97b97cc-rnh5p"]
Apr 22 20:04:21.552704 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.552665    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53185718-efed-4ae5-a2d0-6e0e4301109c-service-ca\") pod \"console-5bd97b97cc-rnh5p\" (UID: \"53185718-efed-4ae5-a2d0-6e0e4301109c\") " pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.552704 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.552701    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53185718-efed-4ae5-a2d0-6e0e4301109c-oauth-serving-cert\") pod \"console-5bd97b97cc-rnh5p\" (UID: \"53185718-efed-4ae5-a2d0-6e0e4301109c\") " pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.552946 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.552742    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53185718-efed-4ae5-a2d0-6e0e4301109c-console-config\") pod \"console-5bd97b97cc-rnh5p\" (UID: \"53185718-efed-4ae5-a2d0-6e0e4301109c\") " pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.552946 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.552792    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6bmn\" (UniqueName: \"kubernetes.io/projected/53185718-efed-4ae5-a2d0-6e0e4301109c-kube-api-access-j6bmn\") pod \"console-5bd97b97cc-rnh5p\" (UID: \"53185718-efed-4ae5-a2d0-6e0e4301109c\") " pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.552946 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.552824    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53185718-efed-4ae5-a2d0-6e0e4301109c-console-oauth-config\") pod \"console-5bd97b97cc-rnh5p\" (UID: \"53185718-efed-4ae5-a2d0-6e0e4301109c\") " pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.552946 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.552876    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53185718-efed-4ae5-a2d0-6e0e4301109c-console-serving-cert\") pod \"console-5bd97b97cc-rnh5p\" (UID: \"53185718-efed-4ae5-a2d0-6e0e4301109c\") " pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.552946 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.552916    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53185718-efed-4ae5-a2d0-6e0e4301109c-trusted-ca-bundle\") pod \"console-5bd97b97cc-rnh5p\" (UID: \"53185718-efed-4ae5-a2d0-6e0e4301109c\") " pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.654492 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.654406    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53185718-efed-4ae5-a2d0-6e0e4301109c-console-serving-cert\") pod \"console-5bd97b97cc-rnh5p\" (UID: \"53185718-efed-4ae5-a2d0-6e0e4301109c\") " pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.654492 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.654451    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53185718-efed-4ae5-a2d0-6e0e4301109c-trusted-ca-bundle\") pod \"console-5bd97b97cc-rnh5p\" (UID: \"53185718-efed-4ae5-a2d0-6e0e4301109c\") " pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.654698 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.654506    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53185718-efed-4ae5-a2d0-6e0e4301109c-service-ca\") pod \"console-5bd97b97cc-rnh5p\" (UID: \"53185718-efed-4ae5-a2d0-6e0e4301109c\") " pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.654698 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.654521    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53185718-efed-4ae5-a2d0-6e0e4301109c-oauth-serving-cert\") pod \"console-5bd97b97cc-rnh5p\" (UID: \"53185718-efed-4ae5-a2d0-6e0e4301109c\") " pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.654698 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.654555    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53185718-efed-4ae5-a2d0-6e0e4301109c-console-config\") pod \"console-5bd97b97cc-rnh5p\" (UID: \"53185718-efed-4ae5-a2d0-6e0e4301109c\") " pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.654698 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.654586    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j6bmn\" (UniqueName: \"kubernetes.io/projected/53185718-efed-4ae5-a2d0-6e0e4301109c-kube-api-access-j6bmn\") pod \"console-5bd97b97cc-rnh5p\" (UID: \"53185718-efed-4ae5-a2d0-6e0e4301109c\") " pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.654698 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.654609    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53185718-efed-4ae5-a2d0-6e0e4301109c-console-oauth-config\") pod \"console-5bd97b97cc-rnh5p\" (UID: \"53185718-efed-4ae5-a2d0-6e0e4301109c\") " pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.655335 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.655306    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53185718-efed-4ae5-a2d0-6e0e4301109c-service-ca\") pod \"console-5bd97b97cc-rnh5p\" (UID: \"53185718-efed-4ae5-a2d0-6e0e4301109c\") " pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.655460 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.655397    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53185718-efed-4ae5-a2d0-6e0e4301109c-console-config\") pod \"console-5bd97b97cc-rnh5p\" (UID: \"53185718-efed-4ae5-a2d0-6e0e4301109c\") " pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.655460 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.655397    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53185718-efed-4ae5-a2d0-6e0e4301109c-oauth-serving-cert\") pod \"console-5bd97b97cc-rnh5p\" (UID: \"53185718-efed-4ae5-a2d0-6e0e4301109c\") " pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.655460 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.655380    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53185718-efed-4ae5-a2d0-6e0e4301109c-trusted-ca-bundle\") pod \"console-5bd97b97cc-rnh5p\" (UID: \"53185718-efed-4ae5-a2d0-6e0e4301109c\") " pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.656904 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.656882    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53185718-efed-4ae5-a2d0-6e0e4301109c-console-oauth-config\") pod \"console-5bd97b97cc-rnh5p\" (UID: \"53185718-efed-4ae5-a2d0-6e0e4301109c\") " pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.657308 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.657293    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53185718-efed-4ae5-a2d0-6e0e4301109c-console-serving-cert\") pod \"console-5bd97b97cc-rnh5p\" (UID: \"53185718-efed-4ae5-a2d0-6e0e4301109c\") " pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.664664 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.664641    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6bmn\" (UniqueName: \"kubernetes.io/projected/53185718-efed-4ae5-a2d0-6e0e4301109c-kube-api-access-j6bmn\") pod \"console-5bd97b97cc-rnh5p\" (UID: \"53185718-efed-4ae5-a2d0-6e0e4301109c\") " pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.695690 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.695661    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:21.872649 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.872309    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bd97b97cc-rnh5p"]
Apr 22 20:04:21.874102 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:04:21.874069    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53185718_efed_4ae5_a2d0_6e0e4301109c.slice/crio-fe78e6c06182eb30a6a43db29924759fcf189051b80eaf111e65f65fbb849676 WatchSource:0}: Error finding container fe78e6c06182eb30a6a43db29924759fcf189051b80eaf111e65f65fbb849676: Status 404 returned error can't find the container with id fe78e6c06182eb30a6a43db29924759fcf189051b80eaf111e65f65fbb849676
Apr 22 20:04:21.978686 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.978647    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bd97b97cc-rnh5p" event={"ID":"53185718-efed-4ae5-a2d0-6e0e4301109c","Type":"ContainerStarted","Data":"74872596f9568bb8c22f8d9f4ca057d8c73f4d5e62a37984c86cf407e81fdc5e"}
Apr 22 20:04:21.978686 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.978688    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bd97b97cc-rnh5p" event={"ID":"53185718-efed-4ae5-a2d0-6e0e4301109c","Type":"ContainerStarted","Data":"fe78e6c06182eb30a6a43db29924759fcf189051b80eaf111e65f65fbb849676"}
Apr 22 20:04:21.997587 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:21.997530    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5bd97b97cc-rnh5p" podStartSLOduration=0.997513879 podStartE2EDuration="997.513879ms" podCreationTimestamp="2026-04-22 20:04:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:04:21.996176417 +0000 UTC m=+397.096132682" watchObservedRunningTime="2026-04-22 20:04:21.997513879 +0000 UTC m=+397.097470143"
Apr 22 20:04:26.232010 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:26.231975    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-khzl8"]
Apr 22 20:04:26.243169 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:26.243131    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-khzl8"
Apr 22 20:04:26.245514 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:26.245484    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-khzl8"]
Apr 22 20:04:26.245680 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:26.245660    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Apr 22 20:04:26.245758 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:26.245666    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Apr 22 20:04:26.245815 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:26.245667    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Apr 22 20:04:26.245815 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:26.245668    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-w92tk\""
Apr 22 20:04:26.398800 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:26.398770    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frscr\" (UniqueName: \"kubernetes.io/projected/302ecd8f-e911-4d5b-b494-b1893a7fc59e-kube-api-access-frscr\") pod \"dns-operator-controller-manager-844548ff4c-khzl8\" (UID: \"302ecd8f-e911-4d5b-b494-b1893a7fc59e\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-khzl8"
Apr 22 20:04:26.499498 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:26.499413    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-frscr\" (UniqueName: \"kubernetes.io/projected/302ecd8f-e911-4d5b-b494-b1893a7fc59e-kube-api-access-frscr\") pod \"dns-operator-controller-manager-844548ff4c-khzl8\" (UID: \"302ecd8f-e911-4d5b-b494-b1893a7fc59e\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-khzl8"
Apr 22 20:04:26.514784 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:26.514752    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frscr\" (UniqueName: \"kubernetes.io/projected/302ecd8f-e911-4d5b-b494-b1893a7fc59e-kube-api-access-frscr\") pod \"dns-operator-controller-manager-844548ff4c-khzl8\" (UID: \"302ecd8f-e911-4d5b-b494-b1893a7fc59e\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-khzl8"
Apr 22 20:04:26.554732 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:26.554702    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-khzl8"
Apr 22 20:04:26.681481 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:26.681447    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-khzl8"]
Apr 22 20:04:26.683802 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:04:26.683776    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod302ecd8f_e911_4d5b_b494_b1893a7fc59e.slice/crio-d9a350eddfbdc2f4856de03ed55e3c070b8f7ea53528f33eba33482477e8bd34 WatchSource:0}: Error finding container d9a350eddfbdc2f4856de03ed55e3c070b8f7ea53528f33eba33482477e8bd34: Status 404 returned error can't find the container with id d9a350eddfbdc2f4856de03ed55e3c070b8f7ea53528f33eba33482477e8bd34
Apr 22 20:04:26.997835 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:26.997797    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-khzl8" event={"ID":"302ecd8f-e911-4d5b-b494-b1893a7fc59e","Type":"ContainerStarted","Data":"d9a350eddfbdc2f4856de03ed55e3c070b8f7ea53528f33eba33482477e8bd34"}
Apr 22 20:04:29.769081 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:29.769044    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-8tvn5"]
Apr 22 20:04:29.773569 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:29.773544    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-8tvn5"
Apr 22 20:04:29.775955 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:29.775931    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tx57d\""
Apr 22 20:04:29.783553 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:29.783529    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-8tvn5"]
Apr 22 20:04:29.931028 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:29.931004    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbgnb\" (UniqueName: \"kubernetes.io/projected/6f2155bb-f25e-4ab1-b66a-77d84bb92aac-kube-api-access-tbgnb\") pod \"kuadrant-operator-controller-manager-6ddf9554fc-8tvn5\" (UID: \"6f2155bb-f25e-4ab1-b66a-77d84bb92aac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-8tvn5"
Apr 22 20:04:29.931165 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:29.931059    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f2155bb-f25e-4ab1-b66a-77d84bb92aac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6ddf9554fc-8tvn5\" (UID: \"6f2155bb-f25e-4ab1-b66a-77d84bb92aac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-8tvn5"
Apr 22 20:04:30.012549 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:30.012513    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-khzl8" event={"ID":"302ecd8f-e911-4d5b-b494-b1893a7fc59e","Type":"ContainerStarted","Data":"3b3e8d80ee8bf123f8a20e8006aea35fdd9bf673301ec1b2c30a0a61638fdfcd"}
Apr 22 20:04:30.012711 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:30.012576    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-khzl8"
Apr 22 20:04:30.032386 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:30.032306    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tbgnb\" (UniqueName: \"kubernetes.io/projected/6f2155bb-f25e-4ab1-b66a-77d84bb92aac-kube-api-access-tbgnb\") pod \"kuadrant-operator-controller-manager-6ddf9554fc-8tvn5\" (UID: \"6f2155bb-f25e-4ab1-b66a-77d84bb92aac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-8tvn5"
Apr 22 20:04:30.032386 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:30.032370    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f2155bb-f25e-4ab1-b66a-77d84bb92aac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6ddf9554fc-8tvn5\" (UID: \"6f2155bb-f25e-4ab1-b66a-77d84bb92aac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-8tvn5"
Apr 22 20:04:30.032637 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:30.032599    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-khzl8" podStartSLOduration=0.857254035 podStartE2EDuration="4.032585802s" podCreationTimestamp="2026-04-22 20:04:26 +0000 UTC" firstStartedPulling="2026-04-22 20:04:26.685828574 +0000 UTC m=+401.785784823" lastFinishedPulling="2026-04-22 20:04:29.861160146 +0000 UTC m=+404.961116590" observedRunningTime="2026-04-22 20:04:30.031001052 +0000 UTC m=+405.130957317" watchObservedRunningTime="2026-04-22 20:04:30.032585802 +0000 UTC m=+405.132542066"
Apr 22 20:04:30.032835 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:30.032811    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f2155bb-f25e-4ab1-b66a-77d84bb92aac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6ddf9554fc-8tvn5\" (UID: \"6f2155bb-f25e-4ab1-b66a-77d84bb92aac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-8tvn5"
Apr 22 20:04:30.046946 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:30.046913    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbgnb\" (UniqueName: \"kubernetes.io/projected/6f2155bb-f25e-4ab1-b66a-77d84bb92aac-kube-api-access-tbgnb\") pod \"kuadrant-operator-controller-manager-6ddf9554fc-8tvn5\" (UID: \"6f2155bb-f25e-4ab1-b66a-77d84bb92aac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-8tvn5"
Apr 22 20:04:30.085611 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:30.085572    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-8tvn5"
Apr 22 20:04:30.224026 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:30.224000    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-8tvn5"]
Apr 22 20:04:30.225586 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:04:30.225560    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2155bb_f25e_4ab1_b66a_77d84bb92aac.slice/crio-da932064b7f5b193d980f94beda1e3fbdee53f71c5bab504a8dcd8c3bc3aca60 WatchSource:0}: Error finding container da932064b7f5b193d980f94beda1e3fbdee53f71c5bab504a8dcd8c3bc3aca60: Status 404 returned error can't find the container with id da932064b7f5b193d980f94beda1e3fbdee53f71c5bab504a8dcd8c3bc3aca60
Apr 22 20:04:31.017449 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:31.017415    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-8tvn5" event={"ID":"6f2155bb-f25e-4ab1-b66a-77d84bb92aac","Type":"ContainerStarted","Data":"da932064b7f5b193d980f94beda1e3fbdee53f71c5bab504a8dcd8c3bc3aca60"}
Apr 22 20:04:31.696485 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:31.696442    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:31.696675 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:31.696496    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:31.702205 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:31.702178    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:32.027264 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:32.027192    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bd97b97cc-rnh5p"
Apr 22 20:04:32.080529 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:32.080492    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-599b7fb674-24kvs"]
Apr 22 20:04:34.444666 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:34.444627    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-skkk9"]
Apr 22 20:04:34.448763 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:34.448736    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-skkk9"
Apr 22 20:04:34.451344 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:34.451309    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-ft9dq\""
Apr 22 20:04:34.457657 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:34.457626    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-skkk9"]
Apr 22 20:04:34.576285 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:34.576240    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmr7c\" (UniqueName: \"kubernetes.io/projected/d733567f-ee6d-4c59-9402-d34f72845cad-kube-api-access-zmr7c\") pod \"limitador-operator-controller-manager-c7fb4c8d5-skkk9\" (UID: \"d733567f-ee6d-4c59-9402-d34f72845cad\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-skkk9"
Apr 22 20:04:34.677216 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:34.677180    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zmr7c\" (UniqueName: \"kubernetes.io/projected/d733567f-ee6d-4c59-9402-d34f72845cad-kube-api-access-zmr7c\") pod \"limitador-operator-controller-manager-c7fb4c8d5-skkk9\" (UID: \"d733567f-ee6d-4c59-9402-d34f72845cad\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-skkk9"
Apr 22 20:04:34.693085 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:34.693047    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmr7c\" (UniqueName: \"kubernetes.io/projected/d733567f-ee6d-4c59-9402-d34f72845cad-kube-api-access-zmr7c\") pod \"limitador-operator-controller-manager-c7fb4c8d5-skkk9\" (UID: \"d733567f-ee6d-4c59-9402-d34f72845cad\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-skkk9"
Apr 22 20:04:34.764542 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:34.764439    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-skkk9"
Apr 22 20:04:35.509335 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:35.509300    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-skkk9"]
Apr 22 20:04:35.511048 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:04:35.511024    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd733567f_ee6d_4c59_9402_d34f72845cad.slice/crio-cdbfd3da29d71d0585882de8f1b028db643665876ef1fbf3c28d53b42ef4e01c WatchSource:0}: Error finding container cdbfd3da29d71d0585882de8f1b028db643665876ef1fbf3c28d53b42ef4e01c: Status 404 returned error can't find the container with id cdbfd3da29d71d0585882de8f1b028db643665876ef1fbf3c28d53b42ef4e01c
Apr 22 20:04:36.045472 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:36.045427    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-8tvn5" event={"ID":"6f2155bb-f25e-4ab1-b66a-77d84bb92aac","Type":"ContainerStarted","Data":"7c1b88263662ce13c50a225675e902ee6997ffbeab308dc0541a7e138e978cc7"}
Apr 22 20:04:36.045669 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:36.045525    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-8tvn5"
Apr 22 20:04:36.046567 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:36.046546    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-skkk9" event={"ID":"d733567f-ee6d-4c59-9402-d34f72845cad","Type":"ContainerStarted","Data":"cdbfd3da29d71d0585882de8f1b028db643665876ef1fbf3c28d53b42ef4e01c"}
Apr 22 20:04:36.066121 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:36.066060    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-8tvn5" podStartSLOduration=1.8338390260000001 podStartE2EDuration="7.066041459s" podCreationTimestamp="2026-04-22 20:04:29 +0000 UTC" firstStartedPulling="2026-04-22 20:04:30.228594614 +0000 UTC m=+405.328550857" lastFinishedPulling="2026-04-22 20:04:35.460797046 +0000 UTC m=+410.560753290" observedRunningTime="2026-04-22 20:04:36.064873502 +0000 UTC m=+411.164829802" watchObservedRunningTime="2026-04-22 20:04:36.066041459 +0000 UTC m=+411.165997726"
Apr 22 20:04:38.055625 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:38.055590    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-skkk9" event={"ID":"d733567f-ee6d-4c59-9402-d34f72845cad","Type":"ContainerStarted","Data":"5e5283204e97ee51c00632b432bc9c17c0414550fbf44a614c5d10dc998130c9"}
Apr 22 20:04:38.055625 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:38.055634    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-skkk9"
Apr 22 20:04:38.072992 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:38.072939    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-skkk9" podStartSLOduration=2.123526589 podStartE2EDuration="4.072924392s" podCreationTimestamp="2026-04-22 20:04:34 +0000 UTC" firstStartedPulling="2026-04-22 20:04:35.513715113 +0000 UTC m=+410.613671355" lastFinishedPulling="2026-04-22 20:04:37.463112915 +0000 UTC m=+412.563069158" observedRunningTime="2026-04-22 20:04:38.070379376 +0000 UTC m=+413.170335640" watchObservedRunningTime="2026-04-22 20:04:38.072924392 +0000 UTC m=+413.172880660"
Apr 22 20:04:41.020312 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:41.020282    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-khzl8"
Apr 22 20:04:47.053325 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:47.053296    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-8tvn5"
Apr 22 20:04:49.062225 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:49.062193    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-skkk9"
Apr 22 20:04:57.108725 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.108661    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-599b7fb674-24kvs" podUID="cf3857b3-1362-4623-a6ab-20a6b7a167cd" containerName="console" containerID="cri-o://eee9d092f1402fa6cd462a39d7cd2f3ef29f9a49ddd7002b6a0c8fff523e2c6c" gracePeriod=15
Apr 22 20:04:57.117984 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.117958    2576 patch_prober.go:28] interesting pod/console-599b7fb674-24kvs container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.18:8443/health\": dial tcp 10.134.0.18:8443: connect: connection refused" start-of-body=
Apr 22 20:04:57.118113 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.118031    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-599b7fb674-24kvs" podUID="cf3857b3-1362-4623-a6ab-20a6b7a167cd" containerName="console" probeResult="failure" output="Get \"https://10.134.0.18:8443/health\": dial tcp 10.134.0.18:8443: connect: connection refused"
Apr 22 20:04:57.351873 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.351849    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-599b7fb674-24kvs_cf3857b3-1362-4623-a6ab-20a6b7a167cd/console/0.log"
Apr 22 20:04:57.352013 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.351927    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:04:57.491740 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.491711    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6b2k\" (UniqueName: \"kubernetes.io/projected/cf3857b3-1362-4623-a6ab-20a6b7a167cd-kube-api-access-w6b2k\") pod \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") "
Apr 22 20:04:57.491900 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.491750    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-oauth-serving-cert\") pod \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") "
Apr 22 20:04:57.491900 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.491767    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-trusted-ca-bundle\") pod \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") "
Apr 22 20:04:57.491900 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.491793    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf3857b3-1362-4623-a6ab-20a6b7a167cd-console-serving-cert\") pod \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") "
Apr 22 20:04:57.491900 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.491826    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cf3857b3-1362-4623-a6ab-20a6b7a167cd-console-oauth-config\") pod \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") "
Apr 22 20:04:57.491900 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.491857    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-service-ca\") pod \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") "
Apr 22 20:04:57.491900 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.491901    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-console-config\") pod \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\" (UID: \"cf3857b3-1362-4623-a6ab-20a6b7a167cd\") "
Apr 22 20:04:57.492219 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.492130    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cf3857b3-1362-4623-a6ab-20a6b7a167cd" (UID: "cf3857b3-1362-4623-a6ab-20a6b7a167cd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:04:57.492281 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.492261    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cf3857b3-1362-4623-a6ab-20a6b7a167cd" (UID: "cf3857b3-1362-4623-a6ab-20a6b7a167cd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:04:57.492481 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.492430    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-service-ca" (OuterVolumeSpecName: "service-ca") pod "cf3857b3-1362-4623-a6ab-20a6b7a167cd" (UID: "cf3857b3-1362-4623-a6ab-20a6b7a167cd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:04:57.492481 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.492468    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-oauth-serving-cert\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:04:57.492595 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.492491    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-trusted-ca-bundle\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:04:57.492595 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.492487    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-console-config" (OuterVolumeSpecName: "console-config") pod "cf3857b3-1362-4623-a6ab-20a6b7a167cd" (UID: "cf3857b3-1362-4623-a6ab-20a6b7a167cd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:04:57.494296 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.494272    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf3857b3-1362-4623-a6ab-20a6b7a167cd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cf3857b3-1362-4623-a6ab-20a6b7a167cd" (UID: "cf3857b3-1362-4623-a6ab-20a6b7a167cd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:04:57.494396 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.494309    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3857b3-1362-4623-a6ab-20a6b7a167cd-kube-api-access-w6b2k" (OuterVolumeSpecName: "kube-api-access-w6b2k") pod "cf3857b3-1362-4623-a6ab-20a6b7a167cd" (UID: "cf3857b3-1362-4623-a6ab-20a6b7a167cd"). InnerVolumeSpecName "kube-api-access-w6b2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:04:57.494396 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.494371    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf3857b3-1362-4623-a6ab-20a6b7a167cd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cf3857b3-1362-4623-a6ab-20a6b7a167cd" (UID: "cf3857b3-1362-4623-a6ab-20a6b7a167cd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:04:57.592868 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.592837    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6b2k\" (UniqueName: \"kubernetes.io/projected/cf3857b3-1362-4623-a6ab-20a6b7a167cd-kube-api-access-w6b2k\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:04:57.592868 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.592864    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf3857b3-1362-4623-a6ab-20a6b7a167cd-console-serving-cert\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:04:57.592868 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.592874    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cf3857b3-1362-4623-a6ab-20a6b7a167cd-console-oauth-config\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:04:57.593074 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.592884    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-service-ca\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:04:57.593074 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:57.592893    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cf3857b3-1362-4623-a6ab-20a6b7a167cd-console-config\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:04:58.140913 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:58.140886    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-599b7fb674-24kvs_cf3857b3-1362-4623-a6ab-20a6b7a167cd/console/0.log"
Apr 22 20:04:58.141312 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:58.140926    2576 generic.go:358] "Generic (PLEG): container finished" podID="cf3857b3-1362-4623-a6ab-20a6b7a167cd" containerID="eee9d092f1402fa6cd462a39d7cd2f3ef29f9a49ddd7002b6a0c8fff523e2c6c" exitCode=2
Apr 22 20:04:58.141312 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:58.140987    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599b7fb674-24kvs"
Apr 22 20:04:58.141312 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:58.141021    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599b7fb674-24kvs" event={"ID":"cf3857b3-1362-4623-a6ab-20a6b7a167cd","Type":"ContainerDied","Data":"eee9d092f1402fa6cd462a39d7cd2f3ef29f9a49ddd7002b6a0c8fff523e2c6c"}
Apr 22 20:04:58.141312 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:58.141059    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599b7fb674-24kvs" event={"ID":"cf3857b3-1362-4623-a6ab-20a6b7a167cd","Type":"ContainerDied","Data":"6d57704d8d122eb67daddcf3cb3c6650ced10e2e1310c4e04ce9eb8199aa4b43"}
Apr 22 20:04:58.141312 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:58.141074    2576 scope.go:117] "RemoveContainer" containerID="eee9d092f1402fa6cd462a39d7cd2f3ef29f9a49ddd7002b6a0c8fff523e2c6c"
Apr 22 20:04:58.150989 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:58.150969    2576 scope.go:117] "RemoveContainer" containerID="eee9d092f1402fa6cd462a39d7cd2f3ef29f9a49ddd7002b6a0c8fff523e2c6c"
Apr 22 20:04:58.151338 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:04:58.151317    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee9d092f1402fa6cd462a39d7cd2f3ef29f9a49ddd7002b6a0c8fff523e2c6c\": container with ID starting with eee9d092f1402fa6cd462a39d7cd2f3ef29f9a49ddd7002b6a0c8fff523e2c6c not found: ID does not exist" containerID="eee9d092f1402fa6cd462a39d7cd2f3ef29f9a49ddd7002b6a0c8fff523e2c6c"
Apr 22 20:04:58.151414 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:58.151346    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee9d092f1402fa6cd462a39d7cd2f3ef29f9a49ddd7002b6a0c8fff523e2c6c"} err="failed to get container status \"eee9d092f1402fa6cd462a39d7cd2f3ef29f9a49ddd7002b6a0c8fff523e2c6c\": rpc error: code = NotFound desc = could not find container \"eee9d092f1402fa6cd462a39d7cd2f3ef29f9a49ddd7002b6a0c8fff523e2c6c\": container with ID starting with eee9d092f1402fa6cd462a39d7cd2f3ef29f9a49ddd7002b6a0c8fff523e2c6c not found: ID does not exist"
Apr 22 20:04:58.167394 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:58.167364    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-599b7fb674-24kvs"]
Apr 22 20:04:58.176592 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:58.176567    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-599b7fb674-24kvs"]
Apr 22 20:04:59.373901 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:04:59.373862    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf3857b3-1362-4623-a6ab-20a6b7a167cd" path="/var/lib/kubelet/pods/cf3857b3-1362-4623-a6ab-20a6b7a167cd/volumes"
Apr 22 20:05:23.086067 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:23.086025    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-p728h"]
Apr 22 20:05:23.086675 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:23.086656    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf3857b3-1362-4623-a6ab-20a6b7a167cd" containerName="console"
Apr 22 20:05:23.086732 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:23.086678    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf3857b3-1362-4623-a6ab-20a6b7a167cd" containerName="console"
Apr 22 20:05:23.086770 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:23.086742    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="cf3857b3-1362-4623-a6ab-20a6b7a167cd" containerName="console"
Apr 22 20:05:23.093866 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:23.093823    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-p728h"
Apr 22 20:05:23.096139 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:23.096106    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ghvlf\""
Apr 22 20:05:23.096279 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:23.096106    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Apr 22 20:05:23.096700 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:23.096675    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-p728h"]
Apr 22 20:05:23.122859 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:23.122825    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-687gd\" (UniqueName: \"kubernetes.io/projected/aae0f147-2a42-4055-99fa-42c4eeb689fa-kube-api-access-687gd\") pod \"limitador-limitador-64c8f475fb-p728h\" (UID: \"aae0f147-2a42-4055-99fa-42c4eeb689fa\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-p728h"
Apr 22 20:05:23.123034 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:23.122890    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aae0f147-2a42-4055-99fa-42c4eeb689fa-config-file\") pod \"limitador-limitador-64c8f475fb-p728h\" (UID: \"aae0f147-2a42-4055-99fa-42c4eeb689fa\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-p728h"
Apr 22 20:05:23.182949 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:23.182904    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-p728h"]
Apr 22 20:05:23.223503 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:23.223471    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-687gd\" (UniqueName: \"kubernetes.io/projected/aae0f147-2a42-4055-99fa-42c4eeb689fa-kube-api-access-687gd\") pod \"limitador-limitador-64c8f475fb-p728h\" (UID: \"aae0f147-2a42-4055-99fa-42c4eeb689fa\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-p728h"
Apr 22 20:05:23.223665 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:23.223541    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aae0f147-2a42-4055-99fa-42c4eeb689fa-config-file\") pod \"limitador-limitador-64c8f475fb-p728h\" (UID: \"aae0f147-2a42-4055-99fa-42c4eeb689fa\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-p728h"
Apr 22 20:05:23.224128 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:23.224109    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aae0f147-2a42-4055-99fa-42c4eeb689fa-config-file\") pod \"limitador-limitador-64c8f475fb-p728h\" (UID: \"aae0f147-2a42-4055-99fa-42c4eeb689fa\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-p728h"
Apr 22 20:05:23.231871 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:23.231846    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-687gd\" (UniqueName: \"kubernetes.io/projected/aae0f147-2a42-4055-99fa-42c4eeb689fa-kube-api-access-687gd\") pod \"limitador-limitador-64c8f475fb-p728h\" (UID: \"aae0f147-2a42-4055-99fa-42c4eeb689fa\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-p728h"
Apr 22 20:05:23.405265 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:23.405229    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-p728h"
Apr 22 20:05:23.545246 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:23.545208    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-p728h"]
Apr 22 20:05:23.547266 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:05:23.547238    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae0f147_2a42_4055_99fa_42c4eeb689fa.slice/crio-d46f805b68d19f92741a8b7a8d944f63d9b82449ffb9dcddb08dbff6d709b115 WatchSource:0}: Error finding container d46f805b68d19f92741a8b7a8d944f63d9b82449ffb9dcddb08dbff6d709b115: Status 404 returned error can't find the container with id d46f805b68d19f92741a8b7a8d944f63d9b82449ffb9dcddb08dbff6d709b115
Apr 22 20:05:24.252086 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:24.252048    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-p728h" event={"ID":"aae0f147-2a42-4055-99fa-42c4eeb689fa","Type":"ContainerStarted","Data":"d46f805b68d19f92741a8b7a8d944f63d9b82449ffb9dcddb08dbff6d709b115"}
Apr 22 20:05:28.276008 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:28.275971    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-p728h" event={"ID":"aae0f147-2a42-4055-99fa-42c4eeb689fa","Type":"ContainerStarted","Data":"c32a62970f24f432bcce21623865d25ce298b60d88fb5e2dce8d3aa63c1e9f0b"}
Apr 22 20:05:28.276008 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:28.276016    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c8f475fb-p728h"
Apr 22 20:05:28.291772 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:28.291710    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c8f475fb-p728h" podStartSLOduration=1.3948064310000001 podStartE2EDuration="5.291691375s" podCreationTimestamp="2026-04-22 20:05:23 +0000 UTC" firstStartedPulling="2026-04-22 20:05:23.549554683 +0000 UTC m=+458.649510926" lastFinishedPulling="2026-04-22 20:05:27.446439626 +0000 UTC m=+462.546395870" observedRunningTime="2026-04-22 20:05:28.289817551 +0000 UTC m=+463.389773815" watchObservedRunningTime="2026-04-22 20:05:28.291691375 +0000 UTC m=+463.391647641"
Apr 22 20:05:38.787206 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:38.787165    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-p728h"]
Apr 22 20:05:38.787624 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:38.787434    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c8f475fb-p728h" podUID="aae0f147-2a42-4055-99fa-42c4eeb689fa" containerName="limitador" containerID="cri-o://c32a62970f24f432bcce21623865d25ce298b60d88fb5e2dce8d3aa63c1e9f0b" gracePeriod=30
Apr 22 20:05:38.788172 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:38.788134    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c8f475fb-p728h"
Apr 22 20:05:39.321838 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:39.321779    2576 generic.go:358] "Generic (PLEG): container finished" podID="aae0f147-2a42-4055-99fa-42c4eeb689fa" containerID="c32a62970f24f432bcce21623865d25ce298b60d88fb5e2dce8d3aa63c1e9f0b" exitCode=0
Apr 22 20:05:39.321985 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:39.321851    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-p728h" event={"ID":"aae0f147-2a42-4055-99fa-42c4eeb689fa","Type":"ContainerDied","Data":"c32a62970f24f432bcce21623865d25ce298b60d88fb5e2dce8d3aa63c1e9f0b"}
Apr 22 20:05:39.321985 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:39.321894    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-p728h" event={"ID":"aae0f147-2a42-4055-99fa-42c4eeb689fa","Type":"ContainerDied","Data":"d46f805b68d19f92741a8b7a8d944f63d9b82449ffb9dcddb08dbff6d709b115"}
Apr 22 20:05:39.321985 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:39.321905    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d46f805b68d19f92741a8b7a8d944f63d9b82449ffb9dcddb08dbff6d709b115"
Apr 22 20:05:39.324628 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:39.324607    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-p728h"
Apr 22 20:05:39.367742 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:39.367695    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aae0f147-2a42-4055-99fa-42c4eeb689fa-config-file\") pod \"aae0f147-2a42-4055-99fa-42c4eeb689fa\" (UID: \"aae0f147-2a42-4055-99fa-42c4eeb689fa\") "
Apr 22 20:05:39.367920 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:39.367802    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-687gd\" (UniqueName: \"kubernetes.io/projected/aae0f147-2a42-4055-99fa-42c4eeb689fa-kube-api-access-687gd\") pod \"aae0f147-2a42-4055-99fa-42c4eeb689fa\" (UID: \"aae0f147-2a42-4055-99fa-42c4eeb689fa\") "
Apr 22 20:05:39.368162 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:39.368119    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aae0f147-2a42-4055-99fa-42c4eeb689fa-config-file" (OuterVolumeSpecName: "config-file") pod "aae0f147-2a42-4055-99fa-42c4eeb689fa" (UID: "aae0f147-2a42-4055-99fa-42c4eeb689fa"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:05:39.370477 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:39.370450    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae0f147-2a42-4055-99fa-42c4eeb689fa-kube-api-access-687gd" (OuterVolumeSpecName: "kube-api-access-687gd") pod "aae0f147-2a42-4055-99fa-42c4eeb689fa" (UID: "aae0f147-2a42-4055-99fa-42c4eeb689fa"). InnerVolumeSpecName "kube-api-access-687gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:05:39.469764 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:39.469725    2576 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aae0f147-2a42-4055-99fa-42c4eeb689fa-config-file\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:05:39.469764 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:39.469764    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-687gd\" (UniqueName: \"kubernetes.io/projected/aae0f147-2a42-4055-99fa-42c4eeb689fa-kube-api-access-687gd\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:05:40.325351 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:40.325318    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-p728h"
Apr 22 20:05:40.340704 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:40.340675    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-p728h"]
Apr 22 20:05:40.344181 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:40.344142    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-p728h"]
Apr 22 20:05:41.374387 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:41.374354    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aae0f147-2a42-4055-99fa-42c4eeb689fa" path="/var/lib/kubelet/pods/aae0f147-2a42-4055-99fa-42c4eeb689fa/volumes"
Apr 22 20:05:59.088870 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.088826    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"]
Apr 22 20:05:59.089479 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.089457    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aae0f147-2a42-4055-99fa-42c4eeb689fa" containerName="limitador"
Apr 22 20:05:59.089479 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.089480    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="aae0f147-2a42-4055-99fa-42c4eeb689fa" containerName="limitador"
Apr 22 20:05:59.089635 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.089563    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="aae0f147-2a42-4055-99fa-42c4eeb689fa" containerName="limitador"
Apr 22 20:05:59.096876 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.096846    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.101436 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.101405    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"]
Apr 22 20:05:59.251302 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.251239    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36f74c3-7c15-47c1-baaa-4043e58c120c-istio-token\") pod \"istiod-openshift-gateway-55ff986f96-92qlr\" (UID: \"a36f74c3-7c15-47c1-baaa-4043e58c120c\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.251302 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.251301    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a36f74c3-7c15-47c1-baaa-4043e58c120c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-55ff986f96-92qlr\" (UID: \"a36f74c3-7c15-47c1-baaa-4043e58c120c\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.251554 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.251345    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36f74c3-7c15-47c1-baaa-4043e58c120c-local-certs\") pod \"istiod-openshift-gateway-55ff986f96-92qlr\" (UID: \"a36f74c3-7c15-47c1-baaa-4043e58c120c\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.251554 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.251442    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a36f74c3-7c15-47c1-baaa-4043e58c120c-istio-kubeconfig\") pod \"istiod-openshift-gateway-55ff986f96-92qlr\" (UID: \"a36f74c3-7c15-47c1-baaa-4043e58c120c\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.251554 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.251466    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a36f74c3-7c15-47c1-baaa-4043e58c120c-cacerts\") pod \"istiod-openshift-gateway-55ff986f96-92qlr\" (UID: \"a36f74c3-7c15-47c1-baaa-4043e58c120c\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.251554 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.251487    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzwkc\" (UniqueName: \"kubernetes.io/projected/a36f74c3-7c15-47c1-baaa-4043e58c120c-kube-api-access-bzwkc\") pod \"istiod-openshift-gateway-55ff986f96-92qlr\" (UID: \"a36f74c3-7c15-47c1-baaa-4043e58c120c\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.251554 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.251514    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a36f74c3-7c15-47c1-baaa-4043e58c120c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-55ff986f96-92qlr\" (UID: \"a36f74c3-7c15-47c1-baaa-4043e58c120c\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.352537 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.352442    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a36f74c3-7c15-47c1-baaa-4043e58c120c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-55ff986f96-92qlr\" (UID: \"a36f74c3-7c15-47c1-baaa-4043e58c120c\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.352537 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.352486    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36f74c3-7c15-47c1-baaa-4043e58c120c-local-certs\") pod \"istiod-openshift-gateway-55ff986f96-92qlr\" (UID: \"a36f74c3-7c15-47c1-baaa-4043e58c120c\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.352787 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.352543    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a36f74c3-7c15-47c1-baaa-4043e58c120c-istio-kubeconfig\") pod \"istiod-openshift-gateway-55ff986f96-92qlr\" (UID: \"a36f74c3-7c15-47c1-baaa-4043e58c120c\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.352787 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.352565    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a36f74c3-7c15-47c1-baaa-4043e58c120c-cacerts\") pod \"istiod-openshift-gateway-55ff986f96-92qlr\" (UID: \"a36f74c3-7c15-47c1-baaa-4043e58c120c\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.352787 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.352593    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bzwkc\" (UniqueName: \"kubernetes.io/projected/a36f74c3-7c15-47c1-baaa-4043e58c120c-kube-api-access-bzwkc\") pod \"istiod-openshift-gateway-55ff986f96-92qlr\" (UID: \"a36f74c3-7c15-47c1-baaa-4043e58c120c\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.352787 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.352622    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a36f74c3-7c15-47c1-baaa-4043e58c120c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-55ff986f96-92qlr\" (UID: \"a36f74c3-7c15-47c1-baaa-4043e58c120c\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.352787 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.352664    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36f74c3-7c15-47c1-baaa-4043e58c120c-istio-token\") pod \"istiod-openshift-gateway-55ff986f96-92qlr\" (UID: \"a36f74c3-7c15-47c1-baaa-4043e58c120c\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.353730 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.353699    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a36f74c3-7c15-47c1-baaa-4043e58c120c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-55ff986f96-92qlr\" (UID: \"a36f74c3-7c15-47c1-baaa-4043e58c120c\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.355363 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.355298    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36f74c3-7c15-47c1-baaa-4043e58c120c-local-certs\") pod \"istiod-openshift-gateway-55ff986f96-92qlr\" (UID: \"a36f74c3-7c15-47c1-baaa-4043e58c120c\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.355685 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.355661    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a36f74c3-7c15-47c1-baaa-4043e58c120c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-55ff986f96-92qlr\" (UID: \"a36f74c3-7c15-47c1-baaa-4043e58c120c\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.355755 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.355661    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a36f74c3-7c15-47c1-baaa-4043e58c120c-cacerts\") pod \"istiod-openshift-gateway-55ff986f96-92qlr\" (UID: \"a36f74c3-7c15-47c1-baaa-4043e58c120c\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.356033 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.356011    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a36f74c3-7c15-47c1-baaa-4043e58c120c-istio-kubeconfig\") pod \"istiod-openshift-gateway-55ff986f96-92qlr\" (UID: \"a36f74c3-7c15-47c1-baaa-4043e58c120c\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.368775 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.368736    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzwkc\" (UniqueName: \"kubernetes.io/projected/a36f74c3-7c15-47c1-baaa-4043e58c120c-kube-api-access-bzwkc\") pod \"istiod-openshift-gateway-55ff986f96-92qlr\" (UID: \"a36f74c3-7c15-47c1-baaa-4043e58c120c\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.387500 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.387456    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36f74c3-7c15-47c1-baaa-4043e58c120c-istio-token\") pod \"istiod-openshift-gateway-55ff986f96-92qlr\" (UID: \"a36f74c3-7c15-47c1-baaa-4043e58c120c\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.408315 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.408275    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:05:59.583333 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.583302    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"]
Apr 22 20:05:59.586075 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:05:59.586042    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36f74c3_7c15_47c1_baaa_4043e58c120c.slice/crio-b71162857b6d443cdf5821af0e888a902c0ccd3bd60300047aeb391d0c6300b1 WatchSource:0}: Error finding container b71162857b6d443cdf5821af0e888a902c0ccd3bd60300047aeb391d0c6300b1: Status 404 returned error can't find the container with id b71162857b6d443cdf5821af0e888a902c0ccd3bd60300047aeb391d0c6300b1
Apr 22 20:05:59.588217 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.588179    2576 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Apr 22 20:05:59.588360 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:05:59.588256    2576 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Apr 22 20:06:00.409081 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.409044    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr" event={"ID":"a36f74c3-7c15-47c1-baaa-4043e58c120c","Type":"ContainerStarted","Data":"a057206d7151da3e7bbc06660bcee8b8a30b6cb3d19b05b1ff9e1afe1360cd08"}
Apr 22 20:06:00.409081 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.409084    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr" event={"ID":"a36f74c3-7c15-47c1-baaa-4043e58c120c","Type":"ContainerStarted","Data":"b71162857b6d443cdf5821af0e888a902c0ccd3bd60300047aeb391d0c6300b1"}
Apr 22 20:06:00.409533 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.409208    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:06:00.410893 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.410869    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr"
Apr 22 20:06:00.485173 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.485084    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-92qlr" podStartSLOduration=1.485064832 podStartE2EDuration="1.485064832s" podCreationTimestamp="2026-04-22 20:05:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:06:00.441990238 +0000 UTC m=+495.541946503" watchObservedRunningTime="2026-04-22 20:06:00.485064832 +0000 UTC m=+495.585021098"
Apr 22 20:06:00.522420 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.522378    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"]
Apr 22 20:06:00.522719 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.522675    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm" podUID="3c2891c8-0663-48d2-8464-4db440e8d0d0" containerName="discovery" containerID="cri-o://9925d66ce299e0b047bace232a5f373c9c91adc33a555678f4f136985631b62b" gracePeriod=30
Apr 22 20:06:00.783617 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.783587    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:06:00.866697 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.866654    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-csr-dns-cert\") pod \"3c2891c8-0663-48d2-8464-4db440e8d0d0\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") "
Apr 22 20:06:00.866697 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.866690    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3c2891c8-0663-48d2-8464-4db440e8d0d0-cacerts\") pod \"3c2891c8-0663-48d2-8464-4db440e8d0d0\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") "
Apr 22 20:06:00.866948 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.866732    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6t4c\" (UniqueName: \"kubernetes.io/projected/3c2891c8-0663-48d2-8464-4db440e8d0d0-kube-api-access-z6t4c\") pod \"3c2891c8-0663-48d2-8464-4db440e8d0d0\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") "
Apr 22 20:06:00.866948 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.866752    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-token\") pod \"3c2891c8-0663-48d2-8464-4db440e8d0d0\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") "
Apr 22 20:06:00.866948 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.866785    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-csr-ca-configmap\") pod \"3c2891c8-0663-48d2-8464-4db440e8d0d0\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") "
Apr 22 20:06:00.866948 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.866833    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2891c8-0663-48d2-8464-4db440e8d0d0-local-certs\") pod \"3c2891c8-0663-48d2-8464-4db440e8d0d0\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") "
Apr 22 20:06:00.866948 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.866873    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-kubeconfig\") pod \"3c2891c8-0663-48d2-8464-4db440e8d0d0\" (UID: \"3c2891c8-0663-48d2-8464-4db440e8d0d0\") "
Apr 22 20:06:00.867268 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.867187    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "3c2891c8-0663-48d2-8464-4db440e8d0d0" (UID: "3c2891c8-0663-48d2-8464-4db440e8d0d0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:06:00.869731 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.869683    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c2891c8-0663-48d2-8464-4db440e8d0d0-cacerts" (OuterVolumeSpecName: "cacerts") pod "3c2891c8-0663-48d2-8464-4db440e8d0d0" (UID: "3c2891c8-0663-48d2-8464-4db440e8d0d0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:06:00.869887 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.869861    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "3c2891c8-0663-48d2-8464-4db440e8d0d0" (UID: "3c2891c8-0663-48d2-8464-4db440e8d0d0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:06:00.870013 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.869976    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2891c8-0663-48d2-8464-4db440e8d0d0-kube-api-access-z6t4c" (OuterVolumeSpecName: "kube-api-access-z6t4c") pod "3c2891c8-0663-48d2-8464-4db440e8d0d0" (UID: "3c2891c8-0663-48d2-8464-4db440e8d0d0"). InnerVolumeSpecName "kube-api-access-z6t4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:06:00.870013 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.869990    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "3c2891c8-0663-48d2-8464-4db440e8d0d0" (UID: "3c2891c8-0663-48d2-8464-4db440e8d0d0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:06:00.870013 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.869993    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2891c8-0663-48d2-8464-4db440e8d0d0-local-certs" (OuterVolumeSpecName: "local-certs") pod "3c2891c8-0663-48d2-8464-4db440e8d0d0" (UID: "3c2891c8-0663-48d2-8464-4db440e8d0d0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:06:00.870139 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.870055    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-token" (OuterVolumeSpecName: "istio-token") pod "3c2891c8-0663-48d2-8464-4db440e8d0d0" (UID: "3c2891c8-0663-48d2-8464-4db440e8d0d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:06:00.968589 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.968487    2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-csr-dns-cert\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:06:00.968589 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.968539    2576 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3c2891c8-0663-48d2-8464-4db440e8d0d0-cacerts\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:06:00.968589 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.968554    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6t4c\" (UniqueName: \"kubernetes.io/projected/3c2891c8-0663-48d2-8464-4db440e8d0d0-kube-api-access-z6t4c\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:06:00.968589 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.968568    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-token\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:06:00.968589 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.968581    2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-csr-ca-configmap\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:06:00.968589 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.968590    2576 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2891c8-0663-48d2-8464-4db440e8d0d0-local-certs\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:06:00.968589 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:00.968599    2576 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c2891c8-0663-48d2-8464-4db440e8d0d0-istio-kubeconfig\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:06:01.415289 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:01.415249    2576 generic.go:358] "Generic (PLEG): container finished" podID="3c2891c8-0663-48d2-8464-4db440e8d0d0" containerID="9925d66ce299e0b047bace232a5f373c9c91adc33a555678f4f136985631b62b" exitCode=0
Apr 22 20:06:01.415702 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:01.415326    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"
Apr 22 20:06:01.415702 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:01.415331    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm" event={"ID":"3c2891c8-0663-48d2-8464-4db440e8d0d0","Type":"ContainerDied","Data":"9925d66ce299e0b047bace232a5f373c9c91adc33a555678f4f136985631b62b"}
Apr 22 20:06:01.415702 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:01.415381    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm" event={"ID":"3c2891c8-0663-48d2-8464-4db440e8d0d0","Type":"ContainerDied","Data":"5a42c94ebd431233a4d2ca73bf7082b561f1e79d83b8d2a4c470eac4acb4dc85"}
Apr 22 20:06:01.415702 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:01.415402    2576 scope.go:117] "RemoveContainer" containerID="9925d66ce299e0b047bace232a5f373c9c91adc33a555678f4f136985631b62b"
Apr 22 20:06:01.424974 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:01.424952    2576 scope.go:117] "RemoveContainer" containerID="9925d66ce299e0b047bace232a5f373c9c91adc33a555678f4f136985631b62b"
Apr 22 20:06:01.425293 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:06:01.425273    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9925d66ce299e0b047bace232a5f373c9c91adc33a555678f4f136985631b62b\": container with ID starting with 9925d66ce299e0b047bace232a5f373c9c91adc33a555678f4f136985631b62b not found: ID does not exist" containerID="9925d66ce299e0b047bace232a5f373c9c91adc33a555678f4f136985631b62b"
Apr 22 20:06:01.425348 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:01.425305    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9925d66ce299e0b047bace232a5f373c9c91adc33a555678f4f136985631b62b"} err="failed to get container status \"9925d66ce299e0b047bace232a5f373c9c91adc33a555678f4f136985631b62b\": rpc error: code = NotFound desc = could not find container \"9925d66ce299e0b047bace232a5f373c9c91adc33a555678f4f136985631b62b\": container with ID starting with 9925d66ce299e0b047bace232a5f373c9c91adc33a555678f4f136985631b62b not found: ID does not exist"
Apr 22 20:06:01.450229 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:01.450185    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"]
Apr 22 20:06:01.456567 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:01.456533    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-v8zwm"]
Apr 22 20:06:03.375511 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:03.375472    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c2891c8-0663-48d2-8464-4db440e8d0d0" path="/var/lib/kubelet/pods/3c2891c8-0663-48d2-8464-4db440e8d0d0/volumes"
Apr 22 20:06:08.112623 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.112579    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-665c47d676-947z6"]
Apr 22 20:06:08.113215 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.113194    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c2891c8-0663-48d2-8464-4db440e8d0d0" containerName="discovery"
Apr 22 20:06:08.113269 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.113221    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c2891c8-0663-48d2-8464-4db440e8d0d0" containerName="discovery"
Apr 22 20:06:08.113349 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.113336    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="3c2891c8-0663-48d2-8464-4db440e8d0d0" containerName="discovery"
Apr 22 20:06:08.116550 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.116529    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-665c47d676-947z6"
Apr 22 20:06:08.119302 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.119276    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Apr 22 20:06:08.119439 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.119405    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Apr 22 20:06:08.119927 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.119906    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-lpcfc\""
Apr 22 20:06:08.123272 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.123250    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Apr 22 20:06:08.128781 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.128250    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-665c47d676-947z6"]
Apr 22 20:06:08.164089 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.164051    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-86cc847c5c-swv2m"]
Apr 22 20:06:08.167370 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.167336    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-86cc847c5c-swv2m"
Apr 22 20:06:08.169738 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.169682    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Apr 22 20:06:08.169865 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.169733    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-8rszl\""
Apr 22 20:06:08.178922 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.178893    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-86cc847c5c-swv2m"]
Apr 22 20:06:08.237484 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.237428    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpnqm\" (UniqueName: \"kubernetes.io/projected/47ece797-b961-449d-ac09-851326e681ac-kube-api-access-dpnqm\") pod \"kserve-controller-manager-665c47d676-947z6\" (UID: \"47ece797-b961-449d-ac09-851326e681ac\") " pod="kserve/kserve-controller-manager-665c47d676-947z6"
Apr 22 20:06:08.237484 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.237485    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47ece797-b961-449d-ac09-851326e681ac-cert\") pod \"kserve-controller-manager-665c47d676-947z6\" (UID: \"47ece797-b961-449d-ac09-851326e681ac\") " pod="kserve/kserve-controller-manager-665c47d676-947z6"
Apr 22 20:06:08.338214 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.338175    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scmhg\" (UniqueName: \"kubernetes.io/projected/39cebb9b-69bf-4cb0-a731-69e9bd9851f2-kube-api-access-scmhg\") pod \"seaweedfs-86cc847c5c-swv2m\" (UID: \"39cebb9b-69bf-4cb0-a731-69e9bd9851f2\") " pod="kserve/seaweedfs-86cc847c5c-swv2m"
Apr 22 20:06:08.338395 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.338221    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/39cebb9b-69bf-4cb0-a731-69e9bd9851f2-data\") pod \"seaweedfs-86cc847c5c-swv2m\" (UID: \"39cebb9b-69bf-4cb0-a731-69e9bd9851f2\") " pod="kserve/seaweedfs-86cc847c5c-swv2m"
Apr 22 20:06:08.338395 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.338300    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dpnqm\" (UniqueName: \"kubernetes.io/projected/47ece797-b961-449d-ac09-851326e681ac-kube-api-access-dpnqm\") pod \"kserve-controller-manager-665c47d676-947z6\" (UID: \"47ece797-b961-449d-ac09-851326e681ac\") " pod="kserve/kserve-controller-manager-665c47d676-947z6"
Apr 22 20:06:08.338395 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.338327    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47ece797-b961-449d-ac09-851326e681ac-cert\") pod \"kserve-controller-manager-665c47d676-947z6\" (UID: \"47ece797-b961-449d-ac09-851326e681ac\") " pod="kserve/kserve-controller-manager-665c47d676-947z6"
Apr 22 20:06:08.338514 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:06:08.338435    2576 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Apr 22 20:06:08.338514 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:06:08.338487    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47ece797-b961-449d-ac09-851326e681ac-cert podName:47ece797-b961-449d-ac09-851326e681ac nodeName:}" failed. No retries permitted until 2026-04-22 20:06:08.838470212 +0000 UTC m=+503.938426455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/47ece797-b961-449d-ac09-851326e681ac-cert") pod "kserve-controller-manager-665c47d676-947z6" (UID: "47ece797-b961-449d-ac09-851326e681ac") : secret "kserve-webhook-server-cert" not found
Apr 22 20:06:08.347068 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.347026    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpnqm\" (UniqueName: \"kubernetes.io/projected/47ece797-b961-449d-ac09-851326e681ac-kube-api-access-dpnqm\") pod \"kserve-controller-manager-665c47d676-947z6\" (UID: \"47ece797-b961-449d-ac09-851326e681ac\") " pod="kserve/kserve-controller-manager-665c47d676-947z6"
Apr 22 20:06:08.439103 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.439058    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-scmhg\" (UniqueName: \"kubernetes.io/projected/39cebb9b-69bf-4cb0-a731-69e9bd9851f2-kube-api-access-scmhg\") pod \"seaweedfs-86cc847c5c-swv2m\" (UID: \"39cebb9b-69bf-4cb0-a731-69e9bd9851f2\") " pod="kserve/seaweedfs-86cc847c5c-swv2m"
Apr 22 20:06:08.439103 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.439103    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/39cebb9b-69bf-4cb0-a731-69e9bd9851f2-data\") pod \"seaweedfs-86cc847c5c-swv2m\" (UID: \"39cebb9b-69bf-4cb0-a731-69e9bd9851f2\") " pod="kserve/seaweedfs-86cc847c5c-swv2m"
Apr 22 20:06:08.439511 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.439491    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/39cebb9b-69bf-4cb0-a731-69e9bd9851f2-data\") pod \"seaweedfs-86cc847c5c-swv2m\" (UID: \"39cebb9b-69bf-4cb0-a731-69e9bd9851f2\") " pod="kserve/seaweedfs-86cc847c5c-swv2m"
Apr 22 20:06:08.447008 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.446983    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scmhg\" (UniqueName: \"kubernetes.io/projected/39cebb9b-69bf-4cb0-a731-69e9bd9851f2-kube-api-access-scmhg\") pod \"seaweedfs-86cc847c5c-swv2m\" (UID: \"39cebb9b-69bf-4cb0-a731-69e9bd9851f2\") " pod="kserve/seaweedfs-86cc847c5c-swv2m"
Apr 22 20:06:08.480346 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.480302    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-86cc847c5c-swv2m"
Apr 22 20:06:08.610527 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.610501    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-86cc847c5c-swv2m"]
Apr 22 20:06:08.612699 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:06:08.612665    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39cebb9b_69bf_4cb0_a731_69e9bd9851f2.slice/crio-664b8b9bd07afcfcc783e9ec913a10980c17274641370a661b86eb4547f93338 WatchSource:0}: Error finding container 664b8b9bd07afcfcc783e9ec913a10980c17274641370a661b86eb4547f93338: Status 404 returned error can't find the container with id 664b8b9bd07afcfcc783e9ec913a10980c17274641370a661b86eb4547f93338
Apr 22 20:06:08.843406 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.843297    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47ece797-b961-449d-ac09-851326e681ac-cert\") pod \"kserve-controller-manager-665c47d676-947z6\" (UID: \"47ece797-b961-449d-ac09-851326e681ac\") " pod="kserve/kserve-controller-manager-665c47d676-947z6"
Apr 22 20:06:08.846083 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:08.846059    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47ece797-b961-449d-ac09-851326e681ac-cert\") pod \"kserve-controller-manager-665c47d676-947z6\" (UID: \"47ece797-b961-449d-ac09-851326e681ac\") " pod="kserve/kserve-controller-manager-665c47d676-947z6"
Apr 22 20:06:09.036297 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:09.036249    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-665c47d676-947z6"
Apr 22 20:06:09.206252 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:09.206204    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-665c47d676-947z6"]
Apr 22 20:06:09.208988 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:06:09.208956    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ece797_b961_449d_ac09_851326e681ac.slice/crio-be9431d7ed8fc94b156fafe3142dfed09c4634e747b32696f4f69b75d2687675 WatchSource:0}: Error finding container be9431d7ed8fc94b156fafe3142dfed09c4634e747b32696f4f69b75d2687675: Status 404 returned error can't find the container with id be9431d7ed8fc94b156fafe3142dfed09c4634e747b32696f4f69b75d2687675
Apr 22 20:06:09.459864 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:09.459822    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-86cc847c5c-swv2m" event={"ID":"39cebb9b-69bf-4cb0-a731-69e9bd9851f2","Type":"ContainerStarted","Data":"664b8b9bd07afcfcc783e9ec913a10980c17274641370a661b86eb4547f93338"}
Apr 22 20:06:09.461015 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:09.460985    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-665c47d676-947z6" event={"ID":"47ece797-b961-449d-ac09-851326e681ac","Type":"ContainerStarted","Data":"be9431d7ed8fc94b156fafe3142dfed09c4634e747b32696f4f69b75d2687675"}
Apr 22 20:06:13.485542 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:13.485500    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-665c47d676-947z6" event={"ID":"47ece797-b961-449d-ac09-851326e681ac","Type":"ContainerStarted","Data":"582c60ff54b77e2170d57e457dcb0da6cd4ec09973fb6971c7590c8b4028c04d"}
Apr 22 20:06:13.486013 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:13.485604    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-665c47d676-947z6"
Apr 22 20:06:13.487008 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:13.486975    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-86cc847c5c-swv2m" event={"ID":"39cebb9b-69bf-4cb0-a731-69e9bd9851f2","Type":"ContainerStarted","Data":"88fef35c0f7b13aa0180b8c9656e57b90d2a12d58e341b65b343993f490f8090"}
Apr 22 20:06:13.487326 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:13.487293    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-86cc847c5c-swv2m"
Apr 22 20:06:13.500400 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:13.500339    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-665c47d676-947z6" podStartSLOduration=2.12100697 podStartE2EDuration="5.500318434s" podCreationTimestamp="2026-04-22 20:06:08 +0000 UTC" firstStartedPulling="2026-04-22 20:06:09.210682414 +0000 UTC m=+504.310638657" lastFinishedPulling="2026-04-22 20:06:12.589993875 +0000 UTC m=+507.689950121" observedRunningTime="2026-04-22 20:06:13.499800556 +0000 UTC m=+508.599756821" watchObservedRunningTime="2026-04-22 20:06:13.500318434 +0000 UTC m=+508.600274717"
Apr 22 20:06:13.516544 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:13.516486    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-86cc847c5c-swv2m" podStartSLOduration=1.435734778 podStartE2EDuration="5.516470395s" podCreationTimestamp="2026-04-22 20:06:08 +0000 UTC" firstStartedPulling="2026-04-22 20:06:08.614008081 +0000 UTC m=+503.713964324" lastFinishedPulling="2026-04-22 20:06:12.694743691 +0000 UTC m=+507.794699941" observedRunningTime="2026-04-22 20:06:13.514119096 +0000 UTC m=+508.614075360" watchObservedRunningTime="2026-04-22 20:06:13.516470395 +0000 UTC m=+508.616426659"
Apr 22 20:06:19.494007 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:19.493969    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-86cc847c5c-swv2m"
Apr 22 20:06:44.498454 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:44.498360    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-665c47d676-947z6"
Apr 22 20:06:45.906462 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:45.906426    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-665c47d676-947z6"]
Apr 22 20:06:45.906867 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:45.906669    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/kserve-controller-manager-665c47d676-947z6" podUID="47ece797-b961-449d-ac09-851326e681ac" containerName="manager" containerID="cri-o://582c60ff54b77e2170d57e457dcb0da6cd4ec09973fb6971c7590c8b4028c04d" gracePeriod=10
Apr 22 20:06:45.927598 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:45.927561    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-665c47d676-tslkg"]
Apr 22 20:06:45.931710 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:45.931691    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-665c47d676-tslkg"
Apr 22 20:06:45.936451 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:45.936422    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-665c47d676-tslkg"]
Apr 22 20:06:46.058253 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.058214    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9eaf6ba0-561f-466e-83f3-ad224f3657dd-cert\") pod \"kserve-controller-manager-665c47d676-tslkg\" (UID: \"9eaf6ba0-561f-466e-83f3-ad224f3657dd\") " pod="kserve/kserve-controller-manager-665c47d676-tslkg"
Apr 22 20:06:46.058438 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.058272    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fs88\" (UniqueName: \"kubernetes.io/projected/9eaf6ba0-561f-466e-83f3-ad224f3657dd-kube-api-access-4fs88\") pod \"kserve-controller-manager-665c47d676-tslkg\" (UID: \"9eaf6ba0-561f-466e-83f3-ad224f3657dd\") " pod="kserve/kserve-controller-manager-665c47d676-tslkg"
Apr 22 20:06:46.155633 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.155607    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-665c47d676-947z6"
Apr 22 20:06:46.159102 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.159028    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9eaf6ba0-561f-466e-83f3-ad224f3657dd-cert\") pod \"kserve-controller-manager-665c47d676-tslkg\" (UID: \"9eaf6ba0-561f-466e-83f3-ad224f3657dd\") " pod="kserve/kserve-controller-manager-665c47d676-tslkg"
Apr 22 20:06:46.159102 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.159075    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4fs88\" (UniqueName: \"kubernetes.io/projected/9eaf6ba0-561f-466e-83f3-ad224f3657dd-kube-api-access-4fs88\") pod \"kserve-controller-manager-665c47d676-tslkg\" (UID: \"9eaf6ba0-561f-466e-83f3-ad224f3657dd\") " pod="kserve/kserve-controller-manager-665c47d676-tslkg"
Apr 22 20:06:46.161582 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.161551    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9eaf6ba0-561f-466e-83f3-ad224f3657dd-cert\") pod \"kserve-controller-manager-665c47d676-tslkg\" (UID: \"9eaf6ba0-561f-466e-83f3-ad224f3657dd\") " pod="kserve/kserve-controller-manager-665c47d676-tslkg"
Apr 22 20:06:46.166089 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.166065    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fs88\" (UniqueName: \"kubernetes.io/projected/9eaf6ba0-561f-466e-83f3-ad224f3657dd-kube-api-access-4fs88\") pod \"kserve-controller-manager-665c47d676-tslkg\" (UID: \"9eaf6ba0-561f-466e-83f3-ad224f3657dd\") " pod="kserve/kserve-controller-manager-665c47d676-tslkg"
Apr 22 20:06:46.260417 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.260374    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47ece797-b961-449d-ac09-851326e681ac-cert\") pod \"47ece797-b961-449d-ac09-851326e681ac\" (UID: \"47ece797-b961-449d-ac09-851326e681ac\") "
Apr 22 20:06:46.260591 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.260440    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpnqm\" (UniqueName: \"kubernetes.io/projected/47ece797-b961-449d-ac09-851326e681ac-kube-api-access-dpnqm\") pod \"47ece797-b961-449d-ac09-851326e681ac\" (UID: \"47ece797-b961-449d-ac09-851326e681ac\") "
Apr 22 20:06:46.262876 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.262837    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ece797-b961-449d-ac09-851326e681ac-cert" (OuterVolumeSpecName: "cert") pod "47ece797-b961-449d-ac09-851326e681ac" (UID: "47ece797-b961-449d-ac09-851326e681ac"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:06:46.262989 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.262883    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ece797-b961-449d-ac09-851326e681ac-kube-api-access-dpnqm" (OuterVolumeSpecName: "kube-api-access-dpnqm") pod "47ece797-b961-449d-ac09-851326e681ac" (UID: "47ece797-b961-449d-ac09-851326e681ac"). InnerVolumeSpecName "kube-api-access-dpnqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:06:46.299754 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.299708    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-665c47d676-tslkg"
Apr 22 20:06:46.362052 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.362020    2576 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47ece797-b961-449d-ac09-851326e681ac-cert\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:06:46.362052 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.362050    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpnqm\" (UniqueName: \"kubernetes.io/projected/47ece797-b961-449d-ac09-851326e681ac-kube-api-access-dpnqm\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:06:46.429779 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.429751    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-665c47d676-tslkg"]
Apr 22 20:06:46.431768 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:06:46.431746    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eaf6ba0_561f_466e_83f3_ad224f3657dd.slice/crio-0531205cbcfbdfac120ff7af1c1a04eb61cf4b0deca8bf87b396a75c65c92cc4 WatchSource:0}: Error finding container 0531205cbcfbdfac120ff7af1c1a04eb61cf4b0deca8bf87b396a75c65c92cc4: Status 404 returned error can't find the container with id 0531205cbcfbdfac120ff7af1c1a04eb61cf4b0deca8bf87b396a75c65c92cc4
Apr 22 20:06:46.626652 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.626617    2576 generic.go:358] "Generic (PLEG): container finished" podID="47ece797-b961-449d-ac09-851326e681ac" containerID="582c60ff54b77e2170d57e457dcb0da6cd4ec09973fb6971c7590c8b4028c04d" exitCode=0
Apr 22 20:06:46.626828 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.626690    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-665c47d676-947z6"
Apr 22 20:06:46.626828 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.626710    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-665c47d676-947z6" event={"ID":"47ece797-b961-449d-ac09-851326e681ac","Type":"ContainerDied","Data":"582c60ff54b77e2170d57e457dcb0da6cd4ec09973fb6971c7590c8b4028c04d"}
Apr 22 20:06:46.626828 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.626757    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-665c47d676-947z6" event={"ID":"47ece797-b961-449d-ac09-851326e681ac","Type":"ContainerDied","Data":"be9431d7ed8fc94b156fafe3142dfed09c4634e747b32696f4f69b75d2687675"}
Apr 22 20:06:46.626828 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.626773    2576 scope.go:117] "RemoveContainer" containerID="582c60ff54b77e2170d57e457dcb0da6cd4ec09973fb6971c7590c8b4028c04d"
Apr 22 20:06:46.628177 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.628128    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-665c47d676-tslkg" event={"ID":"9eaf6ba0-561f-466e-83f3-ad224f3657dd","Type":"ContainerStarted","Data":"0531205cbcfbdfac120ff7af1c1a04eb61cf4b0deca8bf87b396a75c65c92cc4"}
Apr 22 20:06:46.635963 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.635944    2576 scope.go:117] "RemoveContainer" containerID="582c60ff54b77e2170d57e457dcb0da6cd4ec09973fb6971c7590c8b4028c04d"
Apr 22 20:06:46.636304 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:06:46.636284    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"582c60ff54b77e2170d57e457dcb0da6cd4ec09973fb6971c7590c8b4028c04d\": container with ID starting with 582c60ff54b77e2170d57e457dcb0da6cd4ec09973fb6971c7590c8b4028c04d not found: ID does not exist" containerID="582c60ff54b77e2170d57e457dcb0da6cd4ec09973fb6971c7590c8b4028c04d"
Apr 22 20:06:46.636378 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.636329    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"582c60ff54b77e2170d57e457dcb0da6cd4ec09973fb6971c7590c8b4028c04d"} err="failed to get container status \"582c60ff54b77e2170d57e457dcb0da6cd4ec09973fb6971c7590c8b4028c04d\": rpc error: code = NotFound desc = could not find container \"582c60ff54b77e2170d57e457dcb0da6cd4ec09973fb6971c7590c8b4028c04d\": container with ID starting with 582c60ff54b77e2170d57e457dcb0da6cd4ec09973fb6971c7590c8b4028c04d not found: ID does not exist"
Apr 22 20:06:46.648032 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.647996    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-665c47d676-947z6"]
Apr 22 20:06:46.652551 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:46.652520    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-665c47d676-947z6"]
Apr 22 20:06:47.377184 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:47.377134    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47ece797-b961-449d-ac09-851326e681ac" path="/var/lib/kubelet/pods/47ece797-b961-449d-ac09-851326e681ac/volumes"
Apr 22 20:06:47.633808 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:47.633700    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-665c47d676-tslkg" event={"ID":"9eaf6ba0-561f-466e-83f3-ad224f3657dd","Type":"ContainerStarted","Data":"4f2564b53a36bcd792bc0c47542b7c7ede8f9d8d3163ba03a774a3da3f7d745f"}
Apr 22 20:06:47.633808 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:47.633788    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-665c47d676-tslkg"
Apr 22 20:06:47.649049 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:06:47.649000    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-665c47d676-tslkg" podStartSLOduration=2.250691151 podStartE2EDuration="2.648980345s" podCreationTimestamp="2026-04-22 20:06:45 +0000 UTC" firstStartedPulling="2026-04-22 20:06:46.433008943 +0000 UTC m=+541.532965185" lastFinishedPulling="2026-04-22 20:06:46.831298135 +0000 UTC m=+541.931254379" observedRunningTime="2026-04-22 20:06:47.648169676 +0000 UTC m=+542.748125933" watchObservedRunningTime="2026-04-22 20:06:47.648980345 +0000 UTC m=+542.748936609"
Apr 22 20:07:18.643202 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:18.643170    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-665c47d676-tslkg"
Apr 22 20:07:19.503319 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.503282    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-tgxpt"]
Apr 22 20:07:19.503732 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.503717    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47ece797-b961-449d-ac09-851326e681ac" containerName="manager"
Apr 22 20:07:19.503778 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.503734    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="47ece797-b961-449d-ac09-851326e681ac" containerName="manager"
Apr 22 20:07:19.503822 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.503812    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="47ece797-b961-449d-ac09-851326e681ac" containerName="manager"
Apr 22 20:07:19.507050 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.507023    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-tgxpt"
Apr 22 20:07:19.509524 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.509491    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Apr 22 20:07:19.509664 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.509496    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-n5498\""
Apr 22 20:07:19.522042 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.521996    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-tgxpt"]
Apr 22 20:07:19.524778 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.524744    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-2v5hn"]
Apr 22 20:07:19.528836 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.528810    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-2v5hn"
Apr 22 20:07:19.531111 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.531084    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Apr 22 20:07:19.531347 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.531213    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-v4mkg\""
Apr 22 20:07:19.541600 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.541538    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-2v5hn"]
Apr 22 20:07:19.551811 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.551780    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxrmg\" (UniqueName: \"kubernetes.io/projected/1f1a954a-ca38-4aa2-bb51-2a695fae1578-kube-api-access-jxrmg\") pod \"model-serving-api-86f7b4b499-tgxpt\" (UID: \"1f1a954a-ca38-4aa2-bb51-2a695fae1578\") " pod="kserve/model-serving-api-86f7b4b499-tgxpt"
Apr 22 20:07:19.552060 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.551817    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b2bf696-489c-4ee6-8890-43c6b28d60ab-cert\") pod \"odh-model-controller-696fc77849-2v5hn\" (UID: \"9b2bf696-489c-4ee6-8890-43c6b28d60ab\") " pod="kserve/odh-model-controller-696fc77849-2v5hn"
Apr 22 20:07:19.552060 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.551979    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f1a954a-ca38-4aa2-bb51-2a695fae1578-tls-certs\") pod \"model-serving-api-86f7b4b499-tgxpt\" (UID: \"1f1a954a-ca38-4aa2-bb51-2a695fae1578\") " pod="kserve/model-serving-api-86f7b4b499-tgxpt"
Apr 22 20:07:19.552060 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.552014    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcchv\" (UniqueName: \"kubernetes.io/projected/9b2bf696-489c-4ee6-8890-43c6b28d60ab-kube-api-access-vcchv\") pod \"odh-model-controller-696fc77849-2v5hn\" (UID: \"9b2bf696-489c-4ee6-8890-43c6b28d60ab\") " pod="kserve/odh-model-controller-696fc77849-2v5hn"
Apr 22 20:07:19.653053 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.653017    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f1a954a-ca38-4aa2-bb51-2a695fae1578-tls-certs\") pod \"model-serving-api-86f7b4b499-tgxpt\" (UID: \"1f1a954a-ca38-4aa2-bb51-2a695fae1578\") " pod="kserve/model-serving-api-86f7b4b499-tgxpt"
Apr 22 20:07:19.653053 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.653060    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vcchv\" (UniqueName: \"kubernetes.io/projected/9b2bf696-489c-4ee6-8890-43c6b28d60ab-kube-api-access-vcchv\") pod \"odh-model-controller-696fc77849-2v5hn\" (UID: \"9b2bf696-489c-4ee6-8890-43c6b28d60ab\") " pod="kserve/odh-model-controller-696fc77849-2v5hn"
Apr 22 20:07:19.653690 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.653101    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jxrmg\" (UniqueName: \"kubernetes.io/projected/1f1a954a-ca38-4aa2-bb51-2a695fae1578-kube-api-access-jxrmg\") pod \"model-serving-api-86f7b4b499-tgxpt\" (UID: \"1f1a954a-ca38-4aa2-bb51-2a695fae1578\") " pod="kserve/model-serving-api-86f7b4b499-tgxpt"
Apr 22 20:07:19.653690 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.653117    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b2bf696-489c-4ee6-8890-43c6b28d60ab-cert\") pod \"odh-model-controller-696fc77849-2v5hn\" (UID: \"9b2bf696-489c-4ee6-8890-43c6b28d60ab\") " pod="kserve/odh-model-controller-696fc77849-2v5hn"
Apr 22 20:07:19.653690 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:07:19.653224    2576 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Apr 22 20:07:19.653690 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:07:19.653256    2576 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Apr 22 20:07:19.653690 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:07:19.653330    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b2bf696-489c-4ee6-8890-43c6b28d60ab-cert podName:9b2bf696-489c-4ee6-8890-43c6b28d60ab nodeName:}" failed. No retries permitted until 2026-04-22 20:07:20.153296572 +0000 UTC m=+575.253252816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9b2bf696-489c-4ee6-8890-43c6b28d60ab-cert") pod "odh-model-controller-696fc77849-2v5hn" (UID: "9b2bf696-489c-4ee6-8890-43c6b28d60ab") : secret "odh-model-controller-webhook-cert" not found
Apr 22 20:07:19.653690 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:07:19.653359    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f1a954a-ca38-4aa2-bb51-2a695fae1578-tls-certs podName:1f1a954a-ca38-4aa2-bb51-2a695fae1578 nodeName:}" failed. No retries permitted until 2026-04-22 20:07:20.15335027 +0000 UTC m=+575.253306513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/1f1a954a-ca38-4aa2-bb51-2a695fae1578-tls-certs") pod "model-serving-api-86f7b4b499-tgxpt" (UID: "1f1a954a-ca38-4aa2-bb51-2a695fae1578") : secret "model-serving-api-tls" not found
Apr 22 20:07:19.662828 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.662764    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxrmg\" (UniqueName: \"kubernetes.io/projected/1f1a954a-ca38-4aa2-bb51-2a695fae1578-kube-api-access-jxrmg\") pod \"model-serving-api-86f7b4b499-tgxpt\" (UID: \"1f1a954a-ca38-4aa2-bb51-2a695fae1578\") " pod="kserve/model-serving-api-86f7b4b499-tgxpt"
Apr 22 20:07:19.662987 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:19.662837    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcchv\" (UniqueName: \"kubernetes.io/projected/9b2bf696-489c-4ee6-8890-43c6b28d60ab-kube-api-access-vcchv\") pod \"odh-model-controller-696fc77849-2v5hn\" (UID: \"9b2bf696-489c-4ee6-8890-43c6b28d60ab\") " pod="kserve/odh-model-controller-696fc77849-2v5hn"
Apr 22 20:07:20.158234 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:20.158188    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f1a954a-ca38-4aa2-bb51-2a695fae1578-tls-certs\") pod \"model-serving-api-86f7b4b499-tgxpt\" (UID: \"1f1a954a-ca38-4aa2-bb51-2a695fae1578\") " pod="kserve/model-serving-api-86f7b4b499-tgxpt"
Apr 22 20:07:20.158412 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:20.158268    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b2bf696-489c-4ee6-8890-43c6b28d60ab-cert\") pod \"odh-model-controller-696fc77849-2v5hn\" (UID: \"9b2bf696-489c-4ee6-8890-43c6b28d60ab\") " pod="kserve/odh-model-controller-696fc77849-2v5hn"
Apr 22 20:07:20.161066 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:20.161026    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f1a954a-ca38-4aa2-bb51-2a695fae1578-tls-certs\") pod \"model-serving-api-86f7b4b499-tgxpt\" (UID: \"1f1a954a-ca38-4aa2-bb51-2a695fae1578\") " pod="kserve/model-serving-api-86f7b4b499-tgxpt"
Apr 22 20:07:20.161220 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:20.161040    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b2bf696-489c-4ee6-8890-43c6b28d60ab-cert\") pod \"odh-model-controller-696fc77849-2v5hn\" (UID: \"9b2bf696-489c-4ee6-8890-43c6b28d60ab\") " pod="kserve/odh-model-controller-696fc77849-2v5hn"
Apr 22 20:07:20.419404 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:20.419303    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-tgxpt"
Apr 22 20:07:20.452833 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:20.452799    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-2v5hn"
Apr 22 20:07:20.567964 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:20.563114    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-tgxpt"]
Apr 22 20:07:20.601362 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:20.601330    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-2v5hn"]
Apr 22 20:07:20.602997 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:07:20.602969    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2bf696_489c_4ee6_8890_43c6b28d60ab.slice/crio-9ebeb365a3385453b210f6b38984ba2e0a4398d81060a4f154ebd5690b133ced WatchSource:0}: Error finding container 9ebeb365a3385453b210f6b38984ba2e0a4398d81060a4f154ebd5690b133ced: Status 404 returned error can't find the container with id 9ebeb365a3385453b210f6b38984ba2e0a4398d81060a4f154ebd5690b133ced
Apr 22 20:07:20.762866 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:20.762767    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-2v5hn" event={"ID":"9b2bf696-489c-4ee6-8890-43c6b28d60ab","Type":"ContainerStarted","Data":"9ebeb365a3385453b210f6b38984ba2e0a4398d81060a4f154ebd5690b133ced"}
Apr 22 20:07:20.763853 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:20.763816    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-tgxpt" event={"ID":"1f1a954a-ca38-4aa2-bb51-2a695fae1578","Type":"ContainerStarted","Data":"0124758c296911c48950f1ac89be75bf714a44df0a54051e7bbbd99383bdf0df"}
Apr 22 20:07:22.784316 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:22.784274    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-tgxpt" event={"ID":"1f1a954a-ca38-4aa2-bb51-2a695fae1578","Type":"ContainerStarted","Data":"45fc62441813a5dbca848f6f3ab8333d972fe6b66d1cbef08cca128010831783"}
Apr 22 20:07:22.784789 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:22.784406    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-tgxpt"
Apr 22 20:07:22.800372 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:22.800312    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-tgxpt" podStartSLOduration=2.402785326 podStartE2EDuration="3.800295738s" podCreationTimestamp="2026-04-22 20:07:19 +0000 UTC" firstStartedPulling="2026-04-22 20:07:20.572241914 +0000 UTC m=+575.672198157" lastFinishedPulling="2026-04-22 20:07:21.969752323 +0000 UTC m=+577.069708569" observedRunningTime="2026-04-22 20:07:22.798412154 +0000 UTC m=+577.898368425" watchObservedRunningTime="2026-04-22 20:07:22.800295738 +0000 UTC m=+577.900252003"
Apr 22 20:07:23.790675 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:23.790576    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-2v5hn" event={"ID":"9b2bf696-489c-4ee6-8890-43c6b28d60ab","Type":"ContainerStarted","Data":"e3cad60f32fadbe328104fd7babc8e1085091edc88b34576da81d1562ac052bf"}
Apr 22 20:07:23.790675 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:23.790648    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-2v5hn"
Apr 22 20:07:23.806481 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:23.806418    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-2v5hn" podStartSLOduration=2.036622206 podStartE2EDuration="4.806402483s" podCreationTimestamp="2026-04-22 20:07:19 +0000 UTC" firstStartedPulling="2026-04-22 20:07:20.604333102 +0000 UTC m=+575.704289345" lastFinishedPulling="2026-04-22 20:07:23.37411338 +0000 UTC m=+578.474069622" observedRunningTime="2026-04-22 20:07:23.805615864 +0000 UTC m=+578.905572171" watchObservedRunningTime="2026-04-22 20:07:23.806402483 +0000 UTC m=+578.906358747"
Apr 22 20:07:33.795159 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:33.795112    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-tgxpt"
Apr 22 20:07:34.796727 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:34.796687    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-2v5hn"
Apr 22 20:07:35.604068 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:35.604029    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-nghld"]
Apr 22 20:07:35.607367 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:35.607349    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nghld"
Apr 22 20:07:35.615696 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:35.615664    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-nghld"]
Apr 22 20:07:35.707236 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:35.707189    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqvrq\" (UniqueName: \"kubernetes.io/projected/f918f118-912f-463f-a957-a552bbdce85e-kube-api-access-cqvrq\") pod \"s3-init-nghld\" (UID: \"f918f118-912f-463f-a957-a552bbdce85e\") " pod="kserve/s3-init-nghld"
Apr 22 20:07:35.808370 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:35.808324    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cqvrq\" (UniqueName: \"kubernetes.io/projected/f918f118-912f-463f-a957-a552bbdce85e-kube-api-access-cqvrq\") pod \"s3-init-nghld\" (UID: \"f918f118-912f-463f-a957-a552bbdce85e\") " pod="kserve/s3-init-nghld"
Apr 22 20:07:35.817374 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:35.817335    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqvrq\" (UniqueName: \"kubernetes.io/projected/f918f118-912f-463f-a957-a552bbdce85e-kube-api-access-cqvrq\") pod \"s3-init-nghld\" (UID: \"f918f118-912f-463f-a957-a552bbdce85e\") " pod="kserve/s3-init-nghld"
Apr 22 20:07:35.928441 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:35.928403    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nghld"
Apr 22 20:07:36.057792 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:36.057764    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-nghld"]
Apr 22 20:07:36.059861 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:07:36.059834    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf918f118_912f_463f_a957_a552bbdce85e.slice/crio-adad3f8c84643e54672442ae2ea68615318c74328394773183253d9f6e8fddf1 WatchSource:0}: Error finding container adad3f8c84643e54672442ae2ea68615318c74328394773183253d9f6e8fddf1: Status 404 returned error can't find the container with id adad3f8c84643e54672442ae2ea68615318c74328394773183253d9f6e8fddf1
Apr 22 20:07:36.845860 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:36.845820    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nghld" event={"ID":"f918f118-912f-463f-a957-a552bbdce85e","Type":"ContainerStarted","Data":"adad3f8c84643e54672442ae2ea68615318c74328394773183253d9f6e8fddf1"}
Apr 22 20:07:40.865637 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:40.865521    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nghld" event={"ID":"f918f118-912f-463f-a957-a552bbdce85e","Type":"ContainerStarted","Data":"5725e8eb8df1b8f68e0416e1e9c35c499893c2cbccfad1d855db5bac2ec4e714"}
Apr 22 20:07:40.881922 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:40.881853    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-nghld" podStartSLOduration=1.352938124 podStartE2EDuration="5.881837296s" podCreationTimestamp="2026-04-22 20:07:35 +0000 UTC" firstStartedPulling="2026-04-22 20:07:36.06220652 +0000 UTC m=+591.162162766" lastFinishedPulling="2026-04-22 20:07:40.591105694 +0000 UTC m=+595.691061938" observedRunningTime="2026-04-22 20:07:40.879529905 +0000 UTC m=+595.979486171" watchObservedRunningTime="2026-04-22 20:07:40.881837296 +0000 UTC m=+595.981793582"
Apr 22 20:07:43.880396 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:43.880361    2576 generic.go:358] "Generic (PLEG): container finished" podID="f918f118-912f-463f-a957-a552bbdce85e" containerID="5725e8eb8df1b8f68e0416e1e9c35c499893c2cbccfad1d855db5bac2ec4e714" exitCode=0
Apr 22 20:07:43.880774 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:43.880439    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nghld" event={"ID":"f918f118-912f-463f-a957-a552bbdce85e","Type":"ContainerDied","Data":"5725e8eb8df1b8f68e0416e1e9c35c499893c2cbccfad1d855db5bac2ec4e714"}
Apr 22 20:07:45.035198 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:45.035172    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nghld"
Apr 22 20:07:45.196528 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:45.196424    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqvrq\" (UniqueName: \"kubernetes.io/projected/f918f118-912f-463f-a957-a552bbdce85e-kube-api-access-cqvrq\") pod \"f918f118-912f-463f-a957-a552bbdce85e\" (UID: \"f918f118-912f-463f-a957-a552bbdce85e\") "
Apr 22 20:07:45.198675 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:45.198633    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f918f118-912f-463f-a957-a552bbdce85e-kube-api-access-cqvrq" (OuterVolumeSpecName: "kube-api-access-cqvrq") pod "f918f118-912f-463f-a957-a552bbdce85e" (UID: "f918f118-912f-463f-a957-a552bbdce85e"). InnerVolumeSpecName "kube-api-access-cqvrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:07:45.297243 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:45.297200    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqvrq\" (UniqueName: \"kubernetes.io/projected/f918f118-912f-463f-a957-a552bbdce85e-kube-api-access-cqvrq\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:07:45.889870 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:45.889842    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nghld"
Apr 22 20:07:45.890083 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:45.889870    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nghld" event={"ID":"f918f118-912f-463f-a957-a552bbdce85e","Type":"ContainerDied","Data":"adad3f8c84643e54672442ae2ea68615318c74328394773183253d9f6e8fddf1"}
Apr 22 20:07:45.890083 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:45.889906    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adad3f8c84643e54672442ae2ea68615318c74328394773183253d9f6e8fddf1"
Apr 22 20:07:55.941631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:55.941595    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"]
Apr 22 20:07:55.942219 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:55.942199    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f918f118-912f-463f-a957-a552bbdce85e" containerName="s3-init"
Apr 22 20:07:55.942315 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:55.942222    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f918f118-912f-463f-a957-a552bbdce85e" containerName="s3-init"
Apr 22 20:07:55.942368 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:55.942336    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="f918f118-912f-463f-a957-a552bbdce85e" containerName="s3-init"
Apr 22 20:07:55.948644 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:55.948610    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:55.951592 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:55.951563    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 22 20:07:55.951982 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:55.951563    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"istio-ca-root-cert\""
Apr 22 20:07:55.952140 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:55.951673    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Apr 22 20:07:55.952308 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:55.951729    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-mmcsg\""
Apr 22 20:07:55.963510 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:55.963479    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"]
Apr 22 20:07:55.989263 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:55.989224    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-credential-socket\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:55.989263 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:55.989265    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-istio-envoy\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:55.989550 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:55.989289    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:55.989550 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:55.989407    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-workload-certs\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:55.989550 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:55.989474    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-istio-token\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:55.989721 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:55.989529    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-workload-socket\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:55.989721 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:55.989592    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-istio-data\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:55.989721 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:55.989627    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-istio-podinfo\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:55.989721 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:55.989666    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk9pb\" (UniqueName: \"kubernetes.io/projected/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-kube-api-access-qk9pb\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:56.090973 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.090914    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-istio-data\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:56.091211 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.090980    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-istio-podinfo\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:56.091211 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.091018    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qk9pb\" (UniqueName: \"kubernetes.io/projected/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-kube-api-access-qk9pb\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:56.091211 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.091083    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-credential-socket\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:56.091211 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.091118    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-istio-envoy\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:56.091211 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.091177    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:56.091498 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.091235    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-workload-certs\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:56.091498 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.091278    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-istio-token\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:56.091498 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.091309    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-workload-socket\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:56.091498 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.091421    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-istio-data\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:56.091721 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.091675    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-workload-socket\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:56.092189 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.092138    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-workload-certs\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:56.092464 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.092442    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-credential-socket\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:56.092542 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.092491    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:56.094124 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.094096    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-istio-podinfo\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:56.094435 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.094415    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-istio-envoy\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:56.100353 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.100324    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-istio-token\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:56.100843 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.100821    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk9pb\" (UniqueName: \"kubernetes.io/projected/9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4-kube-api-access-qk9pb\") pod \"router-gateway-1-openshift-default-6c59fbf55c-q4nx6\" (UID: \"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:56.265592 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.265494    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:56.427279 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.427250    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"]
Apr 22 20:07:56.430658 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:07:56.430628    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9482a8d9_c8e8_4a15_a8b8_c5f6e351f9e4.slice/crio-25d9cac36d5519d885a4f226358ef7213f369728e580d1044ae8e54d998b4e04 WatchSource:0}: Error finding container 25d9cac36d5519d885a4f226358ef7213f369728e580d1044ae8e54d998b4e04: Status 404 returned error can't find the container with id 25d9cac36d5519d885a4f226358ef7213f369728e580d1044ae8e54d998b4e04
Apr 22 20:07:56.432580 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.432560    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 22 20:07:56.432907 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.432879    2576 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Apr 22 20:07:56.432977 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.432939    2576 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Apr 22 20:07:56.432977 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.432968    2576 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Apr 22 20:07:56.937097 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.937059    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6" event={"ID":"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4","Type":"ContainerStarted","Data":"328045a5af2711cece62ca9a427a8bb10656d6319927ebf7c6cbef74d95d1945"}
Apr 22 20:07:56.937097 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.937095    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6" event={"ID":"9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4","Type":"ContainerStarted","Data":"25d9cac36d5519d885a4f226358ef7213f369728e580d1044ae8e54d998b4e04"}
Apr 22 20:07:56.956238 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:56.956184    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6" podStartSLOduration=1.95616927 podStartE2EDuration="1.95616927s" podCreationTimestamp="2026-04-22 20:07:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:07:56.953870976 +0000 UTC m=+612.053827242" watchObservedRunningTime="2026-04-22 20:07:56.95616927 +0000 UTC m=+612.056125532"
Apr 22 20:07:57.266069 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:57.265975    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:57.271341 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:57.271312    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:57.941593 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:57.941559    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:07:57.942628 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:07:57.942609    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6c59fbf55c-q4nx6"
Apr 22 20:08:14.940219 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:14.940106    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"]
Apr 22 20:08:14.979946 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:14.979905    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"]
Apr 22 20:08:14.980136 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:14.980045    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:14.982863 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:14.982820    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tlcdw\""
Apr 22 20:08:14.983075 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:14.982898    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-mtx69\""
Apr 22 20:08:14.983495 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:14.983471    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Apr 22 20:08:15.068217 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:15.068182    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:15.068431 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:15.068233    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:15.068431 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:15.068303    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:15.068431 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:15.068362    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7bwt\" (UniqueName: \"kubernetes.io/projected/81512f87-0a38-4336-a810-54e5df331605-kube-api-access-s7bwt\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:15.068431 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:15.068407    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:15.068617 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:15.068438    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81512f87-0a38-4336-a810-54e5df331605-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:15.169374 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:15.169329    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:15.169374 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:15.169378    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81512f87-0a38-4336-a810-54e5df331605-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:15.169644 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:15.169438    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:15.169644 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:15.169469    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:15.169644 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:15.169619    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:15.169806 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:15.169693    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s7bwt\" (UniqueName: \"kubernetes.io/projected/81512f87-0a38-4336-a810-54e5df331605-kube-api-access-s7bwt\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:15.169846 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:15.169801    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:15.170048 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:15.170028    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:15.170139 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:15.170059    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:15.170215 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:15.170143    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:15.172335 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:15.172312    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81512f87-0a38-4336-a810-54e5df331605-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:15.178301 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:15.178267    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7bwt\" (UniqueName: \"kubernetes.io/projected/81512f87-0a38-4336-a810-54e5df331605-kube-api-access-s7bwt\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:15.292901 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:15.292797    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:15.452380 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:15.452345    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"]
Apr 22 20:08:15.455483 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:08:15.455444    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81512f87_0a38_4336_a810_54e5df331605.slice/crio-e904c0e7ead8afc78ecf27a1832d0c0fd9b5d80f4f18f6a1e0363032f67ef4df WatchSource:0}: Error finding container e904c0e7ead8afc78ecf27a1832d0c0fd9b5d80f4f18f6a1e0363032f67ef4df: Status 404 returned error can't find the container with id e904c0e7ead8afc78ecf27a1832d0c0fd9b5d80f4f18f6a1e0363032f67ef4df
Apr 22 20:08:16.013722 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:16.013677    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs" event={"ID":"81512f87-0a38-4336-a810-54e5df331605","Type":"ContainerStarted","Data":"e904c0e7ead8afc78ecf27a1832d0c0fd9b5d80f4f18f6a1e0363032f67ef4df"}
Apr 22 20:08:19.030798 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:19.030763    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs" event={"ID":"81512f87-0a38-4336-a810-54e5df331605","Type":"ContainerStarted","Data":"fe5057c16b9d806c0147ac5f67d3512ad1275ba1cabd11c41949c50b7f55cd8a"}
Apr 22 20:08:20.040105 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:20.040009    2576 generic.go:358] "Generic (PLEG): container finished" podID="81512f87-0a38-4336-a810-54e5df331605" containerID="fe5057c16b9d806c0147ac5f67d3512ad1275ba1cabd11c41949c50b7f55cd8a" exitCode=0
Apr 22 20:08:20.040487 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:20.040102    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs" event={"ID":"81512f87-0a38-4336-a810-54e5df331605","Type":"ContainerDied","Data":"fe5057c16b9d806c0147ac5f67d3512ad1275ba1cabd11c41949c50b7f55cd8a"}
Apr 22 20:08:22.053116 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:22.053056    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs" event={"ID":"81512f87-0a38-4336-a810-54e5df331605","Type":"ContainerStarted","Data":"f91e97ed07dd8a30485269b32a65b3398e84f8f51cc6127bcc1d9ceff5fb4627"}
Apr 22 20:08:53.216437 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:53.216398    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs" event={"ID":"81512f87-0a38-4336-a810-54e5df331605","Type":"ContainerStarted","Data":"c9e50ab327f0dc0d45cd552e920905b5906a1544188935fe2e6650fabe17aa86"}
Apr 22 20:08:53.216967 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:53.216598    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:53.226333 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:53.219827    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:53.243487 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:53.243413    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs" podStartSLOduration=2.36842301 podStartE2EDuration="39.243394393s" podCreationTimestamp="2026-04-22 20:08:14 +0000 UTC" firstStartedPulling="2026-04-22 20:08:15.457776286 +0000 UTC m=+630.557732528" lastFinishedPulling="2026-04-22 20:08:52.332747652 +0000 UTC m=+667.432703911" observedRunningTime="2026-04-22 20:08:53.242917448 +0000 UTC m=+668.342873714" watchObservedRunningTime="2026-04-22 20:08:53.243394393 +0000 UTC m=+668.343350658"
Apr 22 20:08:55.293639 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:55.293594    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:08:55.294073 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:08:55.293652    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:09:05.295026 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:05.294988    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:09:05.296253 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:05.296232    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:09:12.636720 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:12.636683    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"]
Apr 22 20:09:12.637317 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:12.637027    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs" podUID="81512f87-0a38-4336-a810-54e5df331605" containerName="main" containerID="cri-o://f91e97ed07dd8a30485269b32a65b3398e84f8f51cc6127bcc1d9ceff5fb4627" gracePeriod=30
Apr 22 20:09:12.637317 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:12.637092    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs" podUID="81512f87-0a38-4336-a810-54e5df331605" containerName="tokenizer" containerID="cri-o://c9e50ab327f0dc0d45cd552e920905b5906a1544188935fe2e6650fabe17aa86" gracePeriod=30
Apr 22 20:09:13.217176 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:09:13.217116    2576 logging.go:55] [core] [Channel #29 SubChannel #30]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.51:9003", ServerName: "10.134.0.51:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.51:9003: connect: connection refused"
Apr 22 20:09:13.311366 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:13.311328    2576 generic.go:358] "Generic (PLEG): container finished" podID="81512f87-0a38-4336-a810-54e5df331605" containerID="f91e97ed07dd8a30485269b32a65b3398e84f8f51cc6127bcc1d9ceff5fb4627" exitCode=0
Apr 22 20:09:13.311541 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:13.311408    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs" event={"ID":"81512f87-0a38-4336-a810-54e5df331605","Type":"ContainerDied","Data":"f91e97ed07dd8a30485269b32a65b3398e84f8f51cc6127bcc1d9ceff5fb4627"}
Apr 22 20:09:13.986088 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:13.986062    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:09:14.160642 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.160592    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-tokenizer-tmp\") pod \"81512f87-0a38-4336-a810-54e5df331605\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") "
Apr 22 20:09:14.160854 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.160681    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-tokenizer-uds\") pod \"81512f87-0a38-4336-a810-54e5df331605\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") "
Apr 22 20:09:14.160854 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.160708    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81512f87-0a38-4336-a810-54e5df331605-tls-certs\") pod \"81512f87-0a38-4336-a810-54e5df331605\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") "
Apr 22 20:09:14.160854 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.160761    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7bwt\" (UniqueName: \"kubernetes.io/projected/81512f87-0a38-4336-a810-54e5df331605-kube-api-access-s7bwt\") pod \"81512f87-0a38-4336-a810-54e5df331605\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") "
Apr 22 20:09:14.160854 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.160791    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-kserve-provision-location\") pod \"81512f87-0a38-4336-a810-54e5df331605\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") "
Apr 22 20:09:14.160854 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.160825    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-tokenizer-cache\") pod \"81512f87-0a38-4336-a810-54e5df331605\" (UID: \"81512f87-0a38-4336-a810-54e5df331605\") "
Apr 22 20:09:14.161127 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.160985    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "81512f87-0a38-4336-a810-54e5df331605" (UID: "81512f87-0a38-4336-a810-54e5df331605"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:09:14.161127 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.160994    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "81512f87-0a38-4336-a810-54e5df331605" (UID: "81512f87-0a38-4336-a810-54e5df331605"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:09:14.161252 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.161136    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "81512f87-0a38-4336-a810-54e5df331605" (UID: "81512f87-0a38-4336-a810-54e5df331605"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:09:14.161535 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.161502    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "81512f87-0a38-4336-a810-54e5df331605" (UID: "81512f87-0a38-4336-a810-54e5df331605"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:09:14.163163 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.163125    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81512f87-0a38-4336-a810-54e5df331605-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81512f87-0a38-4336-a810-54e5df331605" (UID: "81512f87-0a38-4336-a810-54e5df331605"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:09:14.163242 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.163219    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81512f87-0a38-4336-a810-54e5df331605-kube-api-access-s7bwt" (OuterVolumeSpecName: "kube-api-access-s7bwt") pod "81512f87-0a38-4336-a810-54e5df331605" (UID: "81512f87-0a38-4336-a810-54e5df331605"). InnerVolumeSpecName "kube-api-access-s7bwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:09:14.217361 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.217308    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs" podUID="81512f87-0a38-4336-a810-54e5df331605" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.51:9003\" within 1s: context deadline exceeded"
Apr 22 20:09:14.262426 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.262389    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-kserve-provision-location\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:09:14.262426 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.262424    2576 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-tokenizer-cache\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:09:14.262586 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.262436    2576 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-tokenizer-tmp\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:09:14.262586 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.262444    2576 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/81512f87-0a38-4336-a810-54e5df331605-tokenizer-uds\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:09:14.262586 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.262454    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81512f87-0a38-4336-a810-54e5df331605-tls-certs\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:09:14.262586 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.262464    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7bwt\" (UniqueName: \"kubernetes.io/projected/81512f87-0a38-4336-a810-54e5df331605-kube-api-access-s7bwt\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:09:14.317854 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.317815    2576 generic.go:358] "Generic (PLEG): container finished" podID="81512f87-0a38-4336-a810-54e5df331605" containerID="c9e50ab327f0dc0d45cd552e920905b5906a1544188935fe2e6650fabe17aa86" exitCode=0
Apr 22 20:09:14.318021 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.317880    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs" event={"ID":"81512f87-0a38-4336-a810-54e5df331605","Type":"ContainerDied","Data":"c9e50ab327f0dc0d45cd552e920905b5906a1544188935fe2e6650fabe17aa86"}
Apr 22 20:09:14.318021 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.317901    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"
Apr 22 20:09:14.318021 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.317916    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs" event={"ID":"81512f87-0a38-4336-a810-54e5df331605","Type":"ContainerDied","Data":"e904c0e7ead8afc78ecf27a1832d0c0fd9b5d80f4f18f6a1e0363032f67ef4df"}
Apr 22 20:09:14.318021 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.317937    2576 scope.go:117] "RemoveContainer" containerID="c9e50ab327f0dc0d45cd552e920905b5906a1544188935fe2e6650fabe17aa86"
Apr 22 20:09:14.327168 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.327129    2576 scope.go:117] "RemoveContainer" containerID="f91e97ed07dd8a30485269b32a65b3398e84f8f51cc6127bcc1d9ceff5fb4627"
Apr 22 20:09:14.334979 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.334955    2576 scope.go:117] "RemoveContainer" containerID="fe5057c16b9d806c0147ac5f67d3512ad1275ba1cabd11c41949c50b7f55cd8a"
Apr 22 20:09:14.342639 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.342612    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"]
Apr 22 20:09:14.345554 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.345535    2576 scope.go:117] "RemoveContainer" containerID="c9e50ab327f0dc0d45cd552e920905b5906a1544188935fe2e6650fabe17aa86"
Apr 22 20:09:14.345871 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:09:14.345852    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e50ab327f0dc0d45cd552e920905b5906a1544188935fe2e6650fabe17aa86\": container with ID starting with c9e50ab327f0dc0d45cd552e920905b5906a1544188935fe2e6650fabe17aa86 not found: ID does not exist" containerID="c9e50ab327f0dc0d45cd552e920905b5906a1544188935fe2e6650fabe17aa86"
Apr 22 20:09:14.345921 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.345882    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e50ab327f0dc0d45cd552e920905b5906a1544188935fe2e6650fabe17aa86"} err="failed to get container status \"c9e50ab327f0dc0d45cd552e920905b5906a1544188935fe2e6650fabe17aa86\": rpc error: code = NotFound desc = could not find container \"c9e50ab327f0dc0d45cd552e920905b5906a1544188935fe2e6650fabe17aa86\": container with ID starting with c9e50ab327f0dc0d45cd552e920905b5906a1544188935fe2e6650fabe17aa86 not found: ID does not exist"
Apr 22 20:09:14.345921 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.345900    2576 scope.go:117] "RemoveContainer" containerID="f91e97ed07dd8a30485269b32a65b3398e84f8f51cc6127bcc1d9ceff5fb4627"
Apr 22 20:09:14.346192 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:09:14.346175    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91e97ed07dd8a30485269b32a65b3398e84f8f51cc6127bcc1d9ceff5fb4627\": container with ID starting with f91e97ed07dd8a30485269b32a65b3398e84f8f51cc6127bcc1d9ceff5fb4627 not found: ID does not exist" containerID="f91e97ed07dd8a30485269b32a65b3398e84f8f51cc6127bcc1d9ceff5fb4627"
Apr 22 20:09:14.346239 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.346199    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91e97ed07dd8a30485269b32a65b3398e84f8f51cc6127bcc1d9ceff5fb4627"} err="failed to get container status \"f91e97ed07dd8a30485269b32a65b3398e84f8f51cc6127bcc1d9ceff5fb4627\": rpc error: code = NotFound desc = could not find container \"f91e97ed07dd8a30485269b32a65b3398e84f8f51cc6127bcc1d9ceff5fb4627\": container with ID starting with f91e97ed07dd8a30485269b32a65b3398e84f8f51cc6127bcc1d9ceff5fb4627 not found: ID does not exist"
Apr 22 20:09:14.346239 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.346216    2576 scope.go:117] "RemoveContainer" containerID="fe5057c16b9d806c0147ac5f67d3512ad1275ba1cabd11c41949c50b7f55cd8a"
Apr 22 20:09:14.346340 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.346287    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-8784fvjwjs"]
Apr 22 20:09:14.346480 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:09:14.346465    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe5057c16b9d806c0147ac5f67d3512ad1275ba1cabd11c41949c50b7f55cd8a\": container with ID starting with fe5057c16b9d806c0147ac5f67d3512ad1275ba1cabd11c41949c50b7f55cd8a not found: ID does not exist" containerID="fe5057c16b9d806c0147ac5f67d3512ad1275ba1cabd11c41949c50b7f55cd8a"
Apr 22 20:09:14.346531 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:14.346486    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe5057c16b9d806c0147ac5f67d3512ad1275ba1cabd11c41949c50b7f55cd8a"} err="failed to get container status \"fe5057c16b9d806c0147ac5f67d3512ad1275ba1cabd11c41949c50b7f55cd8a\": rpc error: code = NotFound desc = could not find container \"fe5057c16b9d806c0147ac5f67d3512ad1275ba1cabd11c41949c50b7f55cd8a\": container with ID starting with fe5057c16b9d806c0147ac5f67d3512ad1275ba1cabd11c41949c50b7f55cd8a not found: ID does not exist"
Apr 22 20:09:15.374496 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:15.374460    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81512f87-0a38-4336-a810-54e5df331605" path="/var/lib/kubelet/pods/81512f87-0a38-4336-a810-54e5df331605/volumes"
Apr 22 20:09:50.702044 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.701999    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"]
Apr 22 20:09:50.702666 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.702643    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81512f87-0a38-4336-a810-54e5df331605" containerName="storage-initializer"
Apr 22 20:09:50.702733 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.702671    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="81512f87-0a38-4336-a810-54e5df331605" containerName="storage-initializer"
Apr 22 20:09:50.702733 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.702685    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81512f87-0a38-4336-a810-54e5df331605" containerName="main"
Apr 22 20:09:50.702733 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.702694    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="81512f87-0a38-4336-a810-54e5df331605" containerName="main"
Apr 22 20:09:50.702733 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.702709    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81512f87-0a38-4336-a810-54e5df331605" containerName="tokenizer"
Apr 22 20:09:50.702733 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.702717    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="81512f87-0a38-4336-a810-54e5df331605" containerName="tokenizer"
Apr 22 20:09:50.702990 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.702834    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="81512f87-0a38-4336-a810-54e5df331605" containerName="tokenizer"
Apr 22 20:09:50.702990 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.702847    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="81512f87-0a38-4336-a810-54e5df331605" containerName="main"
Apr 22 20:09:50.706840 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.706812    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:09:50.709364 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.709341    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\""
Apr 22 20:09:50.709364 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.709356    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tlcdw\""
Apr 22 20:09:50.716949 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.716587    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"]
Apr 22 20:09:50.814524 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.814491    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:09:50.814709 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.814534    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:09:50.814709 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.814560    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:09:50.814709 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.814586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:09:50.814709 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.814603    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78754072-61b7-40b9-8931-323ca4258271-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:09:50.814856 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.814703    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt2vw\" (UniqueName: \"kubernetes.io/projected/78754072-61b7-40b9-8931-323ca4258271-kube-api-access-xt2vw\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:09:50.908847 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.908810    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"]
Apr 22 20:09:50.912300 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.912267    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:09:50.914675 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.914652    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-sa-dockercfg-s9lzw\""
Apr 22 20:09:50.915292 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.915268    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xt2vw\" (UniqueName: \"kubernetes.io/projected/78754072-61b7-40b9-8931-323ca4258271-kube-api-access-xt2vw\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:09:50.915406 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.915350    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:09:50.915406 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.915389    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:09:50.915525 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.915421    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:09:50.915525 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.915455    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:09:50.915525 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.915481    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78754072-61b7-40b9-8931-323ca4258271-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:09:50.915801 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.915777    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:09:50.915878 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.915797    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:09:50.915878 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.915839    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:09:50.918287 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.918253    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:09:50.918554 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.918535    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78754072-61b7-40b9-8931-323ca4258271-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:09:50.925612 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.925580    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"]
Apr 22 20:09:50.928629 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:50.928598    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt2vw\" (UniqueName: \"kubernetes.io/projected/78754072-61b7-40b9-8931-323ca4258271-kube-api-access-xt2vw\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:09:51.016934 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.016837    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0933abfd-54e7-4bbd-99c3-e72706b3092a-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:09:51.016934 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.016880    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:09:51.016934 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.016909    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:09:51.017176 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.017038    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:09:51.017176 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.017086    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:09:51.017176 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.017138    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svt5b\" (UniqueName: \"kubernetes.io/projected/0933abfd-54e7-4bbd-99c3-e72706b3092a-kube-api-access-svt5b\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:09:51.021347 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.021328    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:09:51.117724 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.117681    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-svt5b\" (UniqueName: \"kubernetes.io/projected/0933abfd-54e7-4bbd-99c3-e72706b3092a-kube-api-access-svt5b\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:09:51.117917 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.117768    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0933abfd-54e7-4bbd-99c3-e72706b3092a-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:09:51.117917 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.117800    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:09:51.117917 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.117829    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:09:51.117917 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.117897    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:09:51.118134 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.117943    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:09:51.118343 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.118305    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:09:51.118488 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.118461    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:09:51.118575 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.118542    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:09:51.118575 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.118562    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:09:51.120999 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.120973    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0933abfd-54e7-4bbd-99c3-e72706b3092a-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:09:51.126351 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.126321    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svt5b\" (UniqueName: \"kubernetes.io/projected/0933abfd-54e7-4bbd-99c3-e72706b3092a-kube-api-access-svt5b\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:09:51.160646 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.160619    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"]
Apr 22 20:09:51.162837 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:09:51.162804    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78754072_61b7_40b9_8931_323ca4258271.slice/crio-dc3156258c38710d26d068b8d94f8e7093016b35cef074d417cc9498b0e235e0 WatchSource:0}: Error finding container dc3156258c38710d26d068b8d94f8e7093016b35cef074d417cc9498b0e235e0: Status 404 returned error can't find the container with id dc3156258c38710d26d068b8d94f8e7093016b35cef074d417cc9498b0e235e0
Apr 22 20:09:51.243976 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.243931    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:09:51.387688 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.387655    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"]
Apr 22 20:09:51.389737 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:09:51.389709    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0933abfd_54e7_4bbd_99c3_e72706b3092a.slice/crio-a3727944824b03f898b959784982aadf4d5179cdb71b2330b6ac7602fb030291 WatchSource:0}: Error finding container a3727944824b03f898b959784982aadf4d5179cdb71b2330b6ac7602fb030291: Status 404 returned error can't find the container with id a3727944824b03f898b959784982aadf4d5179cdb71b2330b6ac7602fb030291
Apr 22 20:09:51.463683 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.463638    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp" event={"ID":"78754072-61b7-40b9-8931-323ca4258271","Type":"ContainerStarted","Data":"58013677f310d5ea4101737d28e4482a4874fa992a1bff716f614be3742999df"}
Apr 22 20:09:51.463683 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.463690    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp" event={"ID":"78754072-61b7-40b9-8931-323ca4258271","Type":"ContainerStarted","Data":"dc3156258c38710d26d068b8d94f8e7093016b35cef074d417cc9498b0e235e0"}
Apr 22 20:09:51.465142 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.465113    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng" event={"ID":"0933abfd-54e7-4bbd-99c3-e72706b3092a","Type":"ContainerStarted","Data":"ea40672d121666d5e415df0512efc2a35646c0ac130ac057475cbbd47029b805"}
Apr 22 20:09:51.465308 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:51.465167    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng" event={"ID":"0933abfd-54e7-4bbd-99c3-e72706b3092a","Type":"ContainerStarted","Data":"a3727944824b03f898b959784982aadf4d5179cdb71b2330b6ac7602fb030291"}
Apr 22 20:09:52.471927 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:52.471890    2576 generic.go:358] "Generic (PLEG): container finished" podID="0933abfd-54e7-4bbd-99c3-e72706b3092a" containerID="ea40672d121666d5e415df0512efc2a35646c0ac130ac057475cbbd47029b805" exitCode=0
Apr 22 20:09:52.472380 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:52.471983    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng" event={"ID":"0933abfd-54e7-4bbd-99c3-e72706b3092a","Type":"ContainerDied","Data":"ea40672d121666d5e415df0512efc2a35646c0ac130ac057475cbbd47029b805"}
Apr 22 20:09:53.479909 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:53.479872    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng" event={"ID":"0933abfd-54e7-4bbd-99c3-e72706b3092a","Type":"ContainerStarted","Data":"64111f3ced7acfc63a2aecb5d104f9439a4bed01f0de786383019f73edd5de15"}
Apr 22 20:09:53.479909 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:53.479913    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng" event={"ID":"0933abfd-54e7-4bbd-99c3-e72706b3092a","Type":"ContainerStarted","Data":"a99b5fddf89eea997b61eac2b6cbd2117cd5db5a45fd833a4c6ebd45b9bf859d"}
Apr 22 20:09:53.480406 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:53.479978    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:09:53.500808 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:53.500751    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng" podStartSLOduration=3.500736865 podStartE2EDuration="3.500736865s" podCreationTimestamp="2026-04-22 20:09:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:09:53.498713864 +0000 UTC m=+728.598670128" watchObservedRunningTime="2026-04-22 20:09:53.500736865 +0000 UTC m=+728.600693128"
Apr 22 20:09:56.495600 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:56.495558    2576 generic.go:358] "Generic (PLEG): container finished" podID="78754072-61b7-40b9-8931-323ca4258271" containerID="58013677f310d5ea4101737d28e4482a4874fa992a1bff716f614be3742999df" exitCode=0
Apr 22 20:09:56.495976 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:09:56.495643    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp" event={"ID":"78754072-61b7-40b9-8931-323ca4258271","Type":"ContainerDied","Data":"58013677f310d5ea4101737d28e4482a4874fa992a1bff716f614be3742999df"}
Apr 22 20:10:01.244430 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:10:01.244270    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:10:01.244922 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:10:01.244481    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:10:01.246467 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:10:01.245938    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng" podUID="0933abfd-54e7-4bbd-99c3-e72706b3092a" containerName="tokenizer" probeResult="failure" output="Get \"http://10.134.0.53:8082/healthz\": dial tcp 10.134.0.53:8082: connect: connection refused"
Apr 22 20:10:11.246915 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:10:11.246882    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:10:11.248245 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:10:11.248218    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:10:24.638982 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:10:24.638945    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp" event={"ID":"78754072-61b7-40b9-8931-323ca4258271","Type":"ContainerStarted","Data":"e1d847453b493f4b757f77cbd4ce93f20d987f15a52865c7e5684c2524222e81"}
Apr 22 20:10:24.662189 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:10:24.660225    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp" podStartSLOduration=7.231752664 podStartE2EDuration="34.660207396s" podCreationTimestamp="2026-04-22 20:09:50 +0000 UTC" firstStartedPulling="2026-04-22 20:09:56.496992091 +0000 UTC m=+731.596948336" lastFinishedPulling="2026-04-22 20:10:23.925446821 +0000 UTC m=+759.025403068" observedRunningTime="2026-04-22 20:10:24.657927855 +0000 UTC m=+759.757884122" watchObservedRunningTime="2026-04-22 20:10:24.660207396 +0000 UTC m=+759.760163664"
Apr 22 20:10:31.021682 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:10:31.021644    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:10:31.022232 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:10:31.021697    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:10:31.023407 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:10:31.023375    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp" podUID="78754072-61b7-40b9-8931-323ca4258271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Apr 22 20:10:31.577648 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:10:31.577611    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:10:41.022398 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:10:41.022341    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp" podUID="78754072-61b7-40b9-8931-323ca4258271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Apr 22 20:10:51.022000 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:10:51.021956    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp" podUID="78754072-61b7-40b9-8931-323ca4258271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Apr 22 20:11:01.021887 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:01.021838    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp" podUID="78754072-61b7-40b9-8931-323ca4258271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Apr 22 20:11:07.401607 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.401505    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"]
Apr 22 20:11:07.407283 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.407247    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:07.412036 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.411972    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Apr 22 20:11:07.417541 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.417500    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"]
Apr 22 20:11:07.587301 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.587265    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p77mj\" (UniqueName: \"kubernetes.io/projected/2876a08a-0944-4f64-af5b-856c71d2f892-kube-api-access-p77mj\") pod \"stop-feature-test-kserve-5bdb8cb8b8-kglzl\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:07.587481 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.587313    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-kserve-provision-location\") pod \"stop-feature-test-kserve-5bdb8cb8b8-kglzl\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:07.587481 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.587335    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2876a08a-0944-4f64-af5b-856c71d2f892-tls-certs\") pod \"stop-feature-test-kserve-5bdb8cb8b8-kglzl\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:07.587481 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.587422    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-dshm\") pod \"stop-feature-test-kserve-5bdb8cb8b8-kglzl\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:07.587481 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.587439    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-model-cache\") pod \"stop-feature-test-kserve-5bdb8cb8b8-kglzl\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:07.587624 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.587496    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-home\") pod \"stop-feature-test-kserve-5bdb8cb8b8-kglzl\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:07.688271 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.688171    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p77mj\" (UniqueName: \"kubernetes.io/projected/2876a08a-0944-4f64-af5b-856c71d2f892-kube-api-access-p77mj\") pod \"stop-feature-test-kserve-5bdb8cb8b8-kglzl\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:07.688271 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.688226    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-kserve-provision-location\") pod \"stop-feature-test-kserve-5bdb8cb8b8-kglzl\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:07.688271 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.688253    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2876a08a-0944-4f64-af5b-856c71d2f892-tls-certs\") pod \"stop-feature-test-kserve-5bdb8cb8b8-kglzl\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:07.688548 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.688312    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-dshm\") pod \"stop-feature-test-kserve-5bdb8cb8b8-kglzl\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:07.688548 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.688531    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-model-cache\") pod \"stop-feature-test-kserve-5bdb8cb8b8-kglzl\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:07.688677 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.688660    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-home\") pod \"stop-feature-test-kserve-5bdb8cb8b8-kglzl\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:07.688792 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.688773    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-model-cache\") pod \"stop-feature-test-kserve-5bdb8cb8b8-kglzl\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:07.688878 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.688853    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-kserve-provision-location\") pod \"stop-feature-test-kserve-5bdb8cb8b8-kglzl\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:07.689030 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.689009    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-home\") pod \"stop-feature-test-kserve-5bdb8cb8b8-kglzl\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:07.691555 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.691529    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-dshm\") pod \"stop-feature-test-kserve-5bdb8cb8b8-kglzl\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:07.692070 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.692040    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2876a08a-0944-4f64-af5b-856c71d2f892-tls-certs\") pod \"stop-feature-test-kserve-5bdb8cb8b8-kglzl\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:07.699076 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.699053    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p77mj\" (UniqueName: \"kubernetes.io/projected/2876a08a-0944-4f64-af5b-856c71d2f892-kube-api-access-p77mj\") pod \"stop-feature-test-kserve-5bdb8cb8b8-kglzl\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:07.723971 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.723939    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:07.861324 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:07.861267    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"]
Apr 22 20:11:07.866062 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:11:07.866034    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2876a08a_0944_4f64_af5b_856c71d2f892.slice/crio-8a4114d1d4cf31bd2989b038d3c63d3e8cf7a55641681c3a5c79f9fc805a9238 WatchSource:0}: Error finding container 8a4114d1d4cf31bd2989b038d3c63d3e8cf7a55641681c3a5c79f9fc805a9238: Status 404 returned error can't find the container with id 8a4114d1d4cf31bd2989b038d3c63d3e8cf7a55641681c3a5c79f9fc805a9238
Apr 22 20:11:08.838026 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:08.837987    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl" event={"ID":"2876a08a-0944-4f64-af5b-856c71d2f892","Type":"ContainerStarted","Data":"fca8dd54f96907cd8dd0530047ea7b80958dc011b76e1ebd8b298851923020ca"}
Apr 22 20:11:08.838026 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:08.838028    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl" event={"ID":"2876a08a-0944-4f64-af5b-856c71d2f892","Type":"ContainerStarted","Data":"8a4114d1d4cf31bd2989b038d3c63d3e8cf7a55641681c3a5c79f9fc805a9238"}
Apr 22 20:11:11.022359 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:11.022312    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp" podUID="78754072-61b7-40b9-8931-323ca4258271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Apr 22 20:11:12.859741 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:12.859704    2576 generic.go:358] "Generic (PLEG): container finished" podID="2876a08a-0944-4f64-af5b-856c71d2f892" containerID="fca8dd54f96907cd8dd0530047ea7b80958dc011b76e1ebd8b298851923020ca" exitCode=0
Apr 22 20:11:12.860134 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:12.859774    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl" event={"ID":"2876a08a-0944-4f64-af5b-856c71d2f892","Type":"ContainerDied","Data":"fca8dd54f96907cd8dd0530047ea7b80958dc011b76e1ebd8b298851923020ca"}
Apr 22 20:11:13.867361 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:13.867317    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl" event={"ID":"2876a08a-0944-4f64-af5b-856c71d2f892","Type":"ContainerStarted","Data":"21978615614c56e86be4fea00575811cce249a1b5b1e96f3dd5dbc9cf7f91075"}
Apr 22 20:11:13.889815 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:13.889746    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl" podStartSLOduration=6.88972583 podStartE2EDuration="6.88972583s" podCreationTimestamp="2026-04-22 20:11:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:11:13.886096748 +0000 UTC m=+808.986053014" watchObservedRunningTime="2026-04-22 20:11:13.88972583 +0000 UTC m=+808.989682096"
Apr 22 20:11:17.724134 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:17.724094    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:17.724134 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:17.724142    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:11:17.725349 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:17.725313    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl" podUID="2876a08a-0944-4f64-af5b-856c71d2f892" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Apr 22 20:11:21.022074 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:21.022019    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp" podUID="78754072-61b7-40b9-8931-323ca4258271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Apr 22 20:11:27.724705 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:27.724659    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl" podUID="2876a08a-0944-4f64-af5b-856c71d2f892" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Apr 22 20:11:31.021934 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:31.021888    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp" podUID="78754072-61b7-40b9-8931-323ca4258271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Apr 22 20:11:37.724355 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:37.724305    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl" podUID="2876a08a-0944-4f64-af5b-856c71d2f892" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Apr 22 20:11:41.022439 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:41.022394    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp" podUID="78754072-61b7-40b9-8931-323ca4258271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Apr 22 20:11:45.387782 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:45.387747    2576 scope.go:117] "RemoveContainer" containerID="c32a62970f24f432bcce21623865d25ce298b60d88fb5e2dce8d3aa63c1e9f0b"
Apr 22 20:11:47.725329 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:47.725288    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl" podUID="2876a08a-0944-4f64-af5b-856c71d2f892" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Apr 22 20:11:51.021762 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:51.021711    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp" podUID="78754072-61b7-40b9-8931-323ca4258271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Apr 22 20:11:57.725021 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:11:57.724970    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl" podUID="2876a08a-0944-4f64-af5b-856c71d2f892" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Apr 22 20:12:01.021947 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:01.021896    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp" podUID="78754072-61b7-40b9-8931-323ca4258271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Apr 22 20:12:07.724661 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:07.724610    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl" podUID="2876a08a-0944-4f64-af5b-856c71d2f892" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Apr 22 20:12:11.032020 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:11.031989    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:12:11.040037 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:11.040009    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:12:17.724617 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:17.724572    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl" podUID="2876a08a-0944-4f64-af5b-856c71d2f892" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Apr 22 20:12:20.215019 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:20.214984    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"]
Apr 22 20:12:20.216097 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:20.215914    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng" podUID="0933abfd-54e7-4bbd-99c3-e72706b3092a" containerName="main" containerID="cri-o://a99b5fddf89eea997b61eac2b6cbd2117cd5db5a45fd833a4c6ebd45b9bf859d" gracePeriod=30
Apr 22 20:12:20.216097 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:20.215957    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng" podUID="0933abfd-54e7-4bbd-99c3-e72706b3092a" containerName="tokenizer" containerID="cri-o://64111f3ced7acfc63a2aecb5d104f9439a4bed01f0de786383019f73edd5de15" gracePeriod=30
Apr 22 20:12:20.219432 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:20.219405    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"]
Apr 22 20:12:20.219881 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:20.219856    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp" podUID="78754072-61b7-40b9-8931-323ca4258271" containerName="main" containerID="cri-o://e1d847453b493f4b757f77cbd4ce93f20d987f15a52865c7e5684c2524222e81" gracePeriod=30
Apr 22 20:12:21.193854 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.193820    2576 generic.go:358] "Generic (PLEG): container finished" podID="0933abfd-54e7-4bbd-99c3-e72706b3092a" containerID="a99b5fddf89eea997b61eac2b6cbd2117cd5db5a45fd833a4c6ebd45b9bf859d" exitCode=0
Apr 22 20:12:21.194060 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.193895    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng" event={"ID":"0933abfd-54e7-4bbd-99c3-e72706b3092a","Type":"ContainerDied","Data":"a99b5fddf89eea997b61eac2b6cbd2117cd5db5a45fd833a4c6ebd45b9bf859d"}
Apr 22 20:12:21.248047 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.247994    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng" podUID="0933abfd-54e7-4bbd-99c3-e72706b3092a" containerName="tokenizer" probeResult="failure" output="Get \"http://10.134.0.53:8082/healthz\": dial tcp 10.134.0.53:8082: connect: connection refused"
Apr 22 20:12:21.579241 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.579214    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:12:21.742734 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.742629    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-tokenizer-tmp\") pod \"0933abfd-54e7-4bbd-99c3-e72706b3092a\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") "
Apr 22 20:12:21.742734 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.742721    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-kserve-provision-location\") pod \"0933abfd-54e7-4bbd-99c3-e72706b3092a\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") "
Apr 22 20:12:21.742979 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.742799    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svt5b\" (UniqueName: \"kubernetes.io/projected/0933abfd-54e7-4bbd-99c3-e72706b3092a-kube-api-access-svt5b\") pod \"0933abfd-54e7-4bbd-99c3-e72706b3092a\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") "
Apr 22 20:12:21.742979 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.742863    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0933abfd-54e7-4bbd-99c3-e72706b3092a-tls-certs\") pod \"0933abfd-54e7-4bbd-99c3-e72706b3092a\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") "
Apr 22 20:12:21.742979 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.742906    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-tokenizer-cache\") pod \"0933abfd-54e7-4bbd-99c3-e72706b3092a\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") "
Apr 22 20:12:21.742979 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.742934    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-tokenizer-uds\") pod \"0933abfd-54e7-4bbd-99c3-e72706b3092a\" (UID: \"0933abfd-54e7-4bbd-99c3-e72706b3092a\") "
Apr 22 20:12:21.743233 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.742999    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "0933abfd-54e7-4bbd-99c3-e72706b3092a" (UID: "0933abfd-54e7-4bbd-99c3-e72706b3092a"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:12:21.743233 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.743217    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "0933abfd-54e7-4bbd-99c3-e72706b3092a" (UID: "0933abfd-54e7-4bbd-99c3-e72706b3092a"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:12:21.743343 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.743287    2576 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-tokenizer-tmp\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:12:21.743343 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.743328    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "0933abfd-54e7-4bbd-99c3-e72706b3092a" (UID: "0933abfd-54e7-4bbd-99c3-e72706b3092a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:12:21.743585 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.743560    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0933abfd-54e7-4bbd-99c3-e72706b3092a" (UID: "0933abfd-54e7-4bbd-99c3-e72706b3092a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:12:21.745042 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.745023    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0933abfd-54e7-4bbd-99c3-e72706b3092a-kube-api-access-svt5b" (OuterVolumeSpecName: "kube-api-access-svt5b") pod "0933abfd-54e7-4bbd-99c3-e72706b3092a" (UID: "0933abfd-54e7-4bbd-99c3-e72706b3092a"). InnerVolumeSpecName "kube-api-access-svt5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:12:21.745172 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.745134    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0933abfd-54e7-4bbd-99c3-e72706b3092a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0933abfd-54e7-4bbd-99c3-e72706b3092a" (UID: "0933abfd-54e7-4bbd-99c3-e72706b3092a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:12:21.844657 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.844614    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-kserve-provision-location\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:12:21.844657 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.844656    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svt5b\" (UniqueName: \"kubernetes.io/projected/0933abfd-54e7-4bbd-99c3-e72706b3092a-kube-api-access-svt5b\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:12:21.844870 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.844676    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0933abfd-54e7-4bbd-99c3-e72706b3092a-tls-certs\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:12:21.844870 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.844689    2576 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-tokenizer-cache\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:12:21.844870 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:21.844699    2576 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0933abfd-54e7-4bbd-99c3-e72706b3092a-tokenizer-uds\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:12:22.199653 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:22.199609    2576 generic.go:358] "Generic (PLEG): container finished" podID="0933abfd-54e7-4bbd-99c3-e72706b3092a" containerID="64111f3ced7acfc63a2aecb5d104f9439a4bed01f0de786383019f73edd5de15" exitCode=0
Apr 22 20:12:22.199890 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:22.199691    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng" event={"ID":"0933abfd-54e7-4bbd-99c3-e72706b3092a","Type":"ContainerDied","Data":"64111f3ced7acfc63a2aecb5d104f9439a4bed01f0de786383019f73edd5de15"}
Apr 22 20:12:22.199890 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:22.199733    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"
Apr 22 20:12:22.199890 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:22.199744    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng" event={"ID":"0933abfd-54e7-4bbd-99c3-e72706b3092a","Type":"ContainerDied","Data":"a3727944824b03f898b959784982aadf4d5179cdb71b2330b6ac7602fb030291"}
Apr 22 20:12:22.199890 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:22.199767    2576 scope.go:117] "RemoveContainer" containerID="64111f3ced7acfc63a2aecb5d104f9439a4bed01f0de786383019f73edd5de15"
Apr 22 20:12:22.210091 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:22.210074    2576 scope.go:117] "RemoveContainer" containerID="a99b5fddf89eea997b61eac2b6cbd2117cd5db5a45fd833a4c6ebd45b9bf859d"
Apr 22 20:12:22.219352 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:22.219331    2576 scope.go:117] "RemoveContainer" containerID="ea40672d121666d5e415df0512efc2a35646c0ac130ac057475cbbd47029b805"
Apr 22 20:12:22.223683 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:22.223652    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"]
Apr 22 20:12:22.227172 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:22.227124    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng"]
Apr 22 20:12:22.229172 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:22.229141    2576 scope.go:117] "RemoveContainer" containerID="64111f3ced7acfc63a2aecb5d104f9439a4bed01f0de786383019f73edd5de15"
Apr 22 20:12:22.229452 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:12:22.229431    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64111f3ced7acfc63a2aecb5d104f9439a4bed01f0de786383019f73edd5de15\": container with ID starting with 64111f3ced7acfc63a2aecb5d104f9439a4bed01f0de786383019f73edd5de15 not found: ID does not exist" containerID="64111f3ced7acfc63a2aecb5d104f9439a4bed01f0de786383019f73edd5de15"
Apr 22 20:12:22.229552 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:22.229458    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64111f3ced7acfc63a2aecb5d104f9439a4bed01f0de786383019f73edd5de15"} err="failed to get container status \"64111f3ced7acfc63a2aecb5d104f9439a4bed01f0de786383019f73edd5de15\": rpc error: code = NotFound desc = could not find container \"64111f3ced7acfc63a2aecb5d104f9439a4bed01f0de786383019f73edd5de15\": container with ID starting with 64111f3ced7acfc63a2aecb5d104f9439a4bed01f0de786383019f73edd5de15 not found: ID does not exist"
Apr 22 20:12:22.229552 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:22.229476    2576 scope.go:117] "RemoveContainer" containerID="a99b5fddf89eea997b61eac2b6cbd2117cd5db5a45fd833a4c6ebd45b9bf859d"
Apr 22 20:12:22.229717 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:12:22.229699    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a99b5fddf89eea997b61eac2b6cbd2117cd5db5a45fd833a4c6ebd45b9bf859d\": container with ID starting with a99b5fddf89eea997b61eac2b6cbd2117cd5db5a45fd833a4c6ebd45b9bf859d not found: ID does not exist" containerID="a99b5fddf89eea997b61eac2b6cbd2117cd5db5a45fd833a4c6ebd45b9bf859d"
Apr 22 20:12:22.229756 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:22.229726    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a99b5fddf89eea997b61eac2b6cbd2117cd5db5a45fd833a4c6ebd45b9bf859d"} err="failed to get container status \"a99b5fddf89eea997b61eac2b6cbd2117cd5db5a45fd833a4c6ebd45b9bf859d\": rpc error: code = NotFound desc = could not find container \"a99b5fddf89eea997b61eac2b6cbd2117cd5db5a45fd833a4c6ebd45b9bf859d\": container with ID starting with a99b5fddf89eea997b61eac2b6cbd2117cd5db5a45fd833a4c6ebd45b9bf859d not found: ID does not exist"
Apr 22 20:12:22.229756 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:22.229740    2576 scope.go:117] "RemoveContainer" containerID="ea40672d121666d5e415df0512efc2a35646c0ac130ac057475cbbd47029b805"
Apr 22 20:12:22.229978 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:12:22.229960    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea40672d121666d5e415df0512efc2a35646c0ac130ac057475cbbd47029b805\": container with ID starting with ea40672d121666d5e415df0512efc2a35646c0ac130ac057475cbbd47029b805 not found: ID does not exist" containerID="ea40672d121666d5e415df0512efc2a35646c0ac130ac057475cbbd47029b805"
Apr 22 20:12:22.230024 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:22.229984    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea40672d121666d5e415df0512efc2a35646c0ac130ac057475cbbd47029b805"} err="failed to get container status \"ea40672d121666d5e415df0512efc2a35646c0ac130ac057475cbbd47029b805\": rpc error: code = NotFound desc = could not find container \"ea40672d121666d5e415df0512efc2a35646c0ac130ac057475cbbd47029b805\": container with ID starting with ea40672d121666d5e415df0512efc2a35646c0ac130ac057475cbbd47029b805 not found: ID does not exist"
Apr 22 20:12:22.576762 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:22.576671    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-sche2n4ng" podUID="0933abfd-54e7-4bbd-99c3-e72706b3092a" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.53:9003\" within 1s: context deadline exceeded"
Apr 22 20:12:22.577110 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:12:22.576777    2576 logging.go:55] [core] [Channel #106 SubChannel #107]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.53:9003", ServerName: "10.134.0.53:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.53:9003: operation was canceled"
Apr 22 20:12:23.375237 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:23.375188    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0933abfd-54e7-4bbd-99c3-e72706b3092a" path="/var/lib/kubelet/pods/0933abfd-54e7-4bbd-99c3-e72706b3092a/volumes"
Apr 22 20:12:26.204443 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.204403    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"]
Apr 22 20:12:26.205108 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.205085    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0933abfd-54e7-4bbd-99c3-e72706b3092a" containerName="storage-initializer"
Apr 22 20:12:26.205178 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.205115    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0933abfd-54e7-4bbd-99c3-e72706b3092a" containerName="storage-initializer"
Apr 22 20:12:26.205178 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.205133    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0933abfd-54e7-4bbd-99c3-e72706b3092a" containerName="tokenizer"
Apr 22 20:12:26.205178 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.205164    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0933abfd-54e7-4bbd-99c3-e72706b3092a" containerName="tokenizer"
Apr 22 20:12:26.205278 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.205186    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0933abfd-54e7-4bbd-99c3-e72706b3092a" containerName="main"
Apr 22 20:12:26.205278 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.205195    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0933abfd-54e7-4bbd-99c3-e72706b3092a" containerName="main"
Apr 22 20:12:26.205346 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.205317    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="0933abfd-54e7-4bbd-99c3-e72706b3092a" containerName="main"
Apr 22 20:12:26.205346 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.205334    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="0933abfd-54e7-4bbd-99c3-e72706b3092a" containerName="tokenizer"
Apr 22 20:12:26.211184 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.211131    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:26.214036 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.214012    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Apr 22 20:12:26.216860 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.216835    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"]
Apr 22 20:12:26.283670 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.283636    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-home\") pod \"custom-route-timeout-test-kserve-5c998f8fc6-rjndf\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:26.283844 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.283684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-5c998f8fc6-rjndf\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:26.283844 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.283806    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9eb4d1-7444-41d4-92e2-19873eda2d66-tls-certs\") pod \"custom-route-timeout-test-kserve-5c998f8fc6-rjndf\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:26.283939 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.283873    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-model-cache\") pod \"custom-route-timeout-test-kserve-5c998f8fc6-rjndf\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:26.283939 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.283911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmlsg\" (UniqueName: \"kubernetes.io/projected/3c9eb4d1-7444-41d4-92e2-19873eda2d66-kube-api-access-vmlsg\") pod \"custom-route-timeout-test-kserve-5c998f8fc6-rjndf\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:26.284020 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.283942    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-dshm\") pod \"custom-route-timeout-test-kserve-5c998f8fc6-rjndf\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:26.384868 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.384836    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9eb4d1-7444-41d4-92e2-19873eda2d66-tls-certs\") pod \"custom-route-timeout-test-kserve-5c998f8fc6-rjndf\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:26.385067 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.384913    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-model-cache\") pod \"custom-route-timeout-test-kserve-5c998f8fc6-rjndf\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:26.385067 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.384942    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vmlsg\" (UniqueName: \"kubernetes.io/projected/3c9eb4d1-7444-41d4-92e2-19873eda2d66-kube-api-access-vmlsg\") pod \"custom-route-timeout-test-kserve-5c998f8fc6-rjndf\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:26.385067 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.384972    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-dshm\") pod \"custom-route-timeout-test-kserve-5c998f8fc6-rjndf\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:26.385272 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.385067    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-home\") pod \"custom-route-timeout-test-kserve-5c998f8fc6-rjndf\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:26.385272 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.385168    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-5c998f8fc6-rjndf\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:26.385401 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.385381    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-model-cache\") pod \"custom-route-timeout-test-kserve-5c998f8fc6-rjndf\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:26.385693 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.385664    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-home\") pod \"custom-route-timeout-test-kserve-5c998f8fc6-rjndf\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:26.385802 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.385708    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-5c998f8fc6-rjndf\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:26.387485 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.387460    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-dshm\") pod \"custom-route-timeout-test-kserve-5c998f8fc6-rjndf\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:26.388366 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.388343    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9eb4d1-7444-41d4-92e2-19873eda2d66-tls-certs\") pod \"custom-route-timeout-test-kserve-5c998f8fc6-rjndf\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:26.392631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.392607    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmlsg\" (UniqueName: \"kubernetes.io/projected/3c9eb4d1-7444-41d4-92e2-19873eda2d66-kube-api-access-vmlsg\") pod \"custom-route-timeout-test-kserve-5c998f8fc6-rjndf\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:26.545754 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.545671    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:26.688056 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:26.688024    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"]
Apr 22 20:12:26.688827 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:12:26.688791    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9eb4d1_7444_41d4_92e2_19873eda2d66.slice/crio-4724dce6b93687bebab7f58d497bb9664d5142bcb9d4b81e1762032ee622d071 WatchSource:0}: Error finding container 4724dce6b93687bebab7f58d497bb9664d5142bcb9d4b81e1762032ee622d071: Status 404 returned error can't find the container with id 4724dce6b93687bebab7f58d497bb9664d5142bcb9d4b81e1762032ee622d071
Apr 22 20:12:27.224870 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:27.224832    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf" event={"ID":"3c9eb4d1-7444-41d4-92e2-19873eda2d66","Type":"ContainerStarted","Data":"fc8b457938f49550a77b8f88637e54c1057cb634c5a69343a8a4cef6c6f5a33c"}
Apr 22 20:12:27.224870 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:27.224878    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf" event={"ID":"3c9eb4d1-7444-41d4-92e2-19873eda2d66","Type":"ContainerStarted","Data":"4724dce6b93687bebab7f58d497bb9664d5142bcb9d4b81e1762032ee622d071"}
Apr 22 20:12:27.724879 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:27.724832    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl" podUID="2876a08a-0944-4f64-af5b-856c71d2f892" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Apr 22 20:12:32.250084 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:32.249994    2576 generic.go:358] "Generic (PLEG): container finished" podID="3c9eb4d1-7444-41d4-92e2-19873eda2d66" containerID="fc8b457938f49550a77b8f88637e54c1057cb634c5a69343a8a4cef6c6f5a33c" exitCode=0
Apr 22 20:12:32.250084 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:32.250068    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf" event={"ID":"3c9eb4d1-7444-41d4-92e2-19873eda2d66","Type":"ContainerDied","Data":"fc8b457938f49550a77b8f88637e54c1057cb634c5a69343a8a4cef6c6f5a33c"}
Apr 22 20:12:33.260855 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:33.260804    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf" event={"ID":"3c9eb4d1-7444-41d4-92e2-19873eda2d66","Type":"ContainerStarted","Data":"b7ced1d02c530a165bcbb5989034a6a569a3e19f9b8215d41e7721d55b5014cf"}
Apr 22 20:12:33.281504 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:33.281437    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf" podStartSLOduration=7.281417888 podStartE2EDuration="7.281417888s" podCreationTimestamp="2026-04-22 20:12:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:12:33.279193381 +0000 UTC m=+888.379149646" watchObservedRunningTime="2026-04-22 20:12:33.281417888 +0000 UTC m=+888.381374154"
Apr 22 20:12:36.546170 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:36.546120    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:36.546596 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:36.546199    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:12:36.548016 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:36.547987    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf" podUID="3c9eb4d1-7444-41d4-92e2-19873eda2d66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Apr 22 20:12:37.725266 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:37.725218    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl" podUID="2876a08a-0944-4f64-af5b-856c71d2f892" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Apr 22 20:12:46.547241 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:46.547179    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf" podUID="3c9eb4d1-7444-41d4-92e2-19873eda2d66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Apr 22 20:12:47.725300 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:47.725257    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl" podUID="2876a08a-0944-4f64-af5b-856c71d2f892" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Apr 22 20:12:50.655679 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:50.655649    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp_78754072-61b7-40b9-8931-323ca4258271/main/0.log"
Apr 22 20:12:50.656219 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:50.656082    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:12:50.743024 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:50.742913    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-home\") pod \"78754072-61b7-40b9-8931-323ca4258271\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") "
Apr 22 20:12:50.743024 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:50.743010    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-kserve-provision-location\") pod \"78754072-61b7-40b9-8931-323ca4258271\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") "
Apr 22 20:12:50.743318 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:50.743095    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78754072-61b7-40b9-8931-323ca4258271-tls-certs\") pod \"78754072-61b7-40b9-8931-323ca4258271\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") "
Apr 22 20:12:50.743318 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:50.743119    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xt2vw\" (UniqueName: \"kubernetes.io/projected/78754072-61b7-40b9-8931-323ca4258271-kube-api-access-xt2vw\") pod \"78754072-61b7-40b9-8931-323ca4258271\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") "
Apr 22 20:12:50.743318 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:50.743143    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-dshm\") pod \"78754072-61b7-40b9-8931-323ca4258271\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") "
Apr 22 20:12:50.743318 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:50.743185    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-model-cache\") pod \"78754072-61b7-40b9-8931-323ca4258271\" (UID: \"78754072-61b7-40b9-8931-323ca4258271\") "
Apr 22 20:12:50.743517 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:50.743362    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-home" (OuterVolumeSpecName: "home") pod "78754072-61b7-40b9-8931-323ca4258271" (UID: "78754072-61b7-40b9-8931-323ca4258271"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:12:50.743517 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:50.743488    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-home\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:12:50.743757 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:50.743731    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-model-cache" (OuterVolumeSpecName: "model-cache") pod "78754072-61b7-40b9-8931-323ca4258271" (UID: "78754072-61b7-40b9-8931-323ca4258271"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:12:50.745618 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:50.745577    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78754072-61b7-40b9-8931-323ca4258271-kube-api-access-xt2vw" (OuterVolumeSpecName: "kube-api-access-xt2vw") pod "78754072-61b7-40b9-8931-323ca4258271" (UID: "78754072-61b7-40b9-8931-323ca4258271"). InnerVolumeSpecName "kube-api-access-xt2vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:12:50.746062 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:50.746033    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-dshm" (OuterVolumeSpecName: "dshm") pod "78754072-61b7-40b9-8931-323ca4258271" (UID: "78754072-61b7-40b9-8931-323ca4258271"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:12:50.746062 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:50.746031    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78754072-61b7-40b9-8931-323ca4258271-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "78754072-61b7-40b9-8931-323ca4258271" (UID: "78754072-61b7-40b9-8931-323ca4258271"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:12:50.768131 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:50.768073    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "78754072-61b7-40b9-8931-323ca4258271" (UID: "78754072-61b7-40b9-8931-323ca4258271"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:12:50.844608 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:50.844558    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78754072-61b7-40b9-8931-323ca4258271-tls-certs\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:12:50.844608 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:50.844609    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt2vw\" (UniqueName: \"kubernetes.io/projected/78754072-61b7-40b9-8931-323ca4258271-kube-api-access-xt2vw\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:12:50.844817 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:50.844627    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-dshm\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:12:50.844817 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:50.844642    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-model-cache\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:12:50.844817 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:50.844658    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78754072-61b7-40b9-8931-323ca4258271-kserve-provision-location\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:12:51.343789 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:51.343760    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp_78754072-61b7-40b9-8931-323ca4258271/main/0.log"
Apr 22 20:12:51.344162 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:51.344124    2576 generic.go:358] "Generic (PLEG): container finished" podID="78754072-61b7-40b9-8931-323ca4258271" containerID="e1d847453b493f4b757f77cbd4ce93f20d987f15a52865c7e5684c2524222e81" exitCode=137
Apr 22 20:12:51.344240 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:51.344222    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"
Apr 22 20:12:51.344316 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:51.344223    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp" event={"ID":"78754072-61b7-40b9-8931-323ca4258271","Type":"ContainerDied","Data":"e1d847453b493f4b757f77cbd4ce93f20d987f15a52865c7e5684c2524222e81"}
Apr 22 20:12:51.344372 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:51.344339    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp" event={"ID":"78754072-61b7-40b9-8931-323ca4258271","Type":"ContainerDied","Data":"dc3156258c38710d26d068b8d94f8e7093016b35cef074d417cc9498b0e235e0"}
Apr 22 20:12:51.344372 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:51.344360    2576 scope.go:117] "RemoveContainer" containerID="e1d847453b493f4b757f77cbd4ce93f20d987f15a52865c7e5684c2524222e81"
Apr 22 20:12:51.379358 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:51.379311    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"]
Apr 22 20:12:51.379358 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:51.379358    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-57b5445dc8vzgpp"]
Apr 22 20:12:51.379766 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:51.379729    2576 scope.go:117] "RemoveContainer" containerID="58013677f310d5ea4101737d28e4482a4874fa992a1bff716f614be3742999df"
Apr 22 20:12:51.424068 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:51.424035    2576 scope.go:117] "RemoveContainer" containerID="e1d847453b493f4b757f77cbd4ce93f20d987f15a52865c7e5684c2524222e81"
Apr 22 20:12:51.424501 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:12:51.424450    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1d847453b493f4b757f77cbd4ce93f20d987f15a52865c7e5684c2524222e81\": container with ID starting with e1d847453b493f4b757f77cbd4ce93f20d987f15a52865c7e5684c2524222e81 not found: ID does not exist" containerID="e1d847453b493f4b757f77cbd4ce93f20d987f15a52865c7e5684c2524222e81"
Apr 22 20:12:51.424651 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:51.424509    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1d847453b493f4b757f77cbd4ce93f20d987f15a52865c7e5684c2524222e81"} err="failed to get container status \"e1d847453b493f4b757f77cbd4ce93f20d987f15a52865c7e5684c2524222e81\": rpc error: code = NotFound desc = could not find container \"e1d847453b493f4b757f77cbd4ce93f20d987f15a52865c7e5684c2524222e81\": container with ID starting with e1d847453b493f4b757f77cbd4ce93f20d987f15a52865c7e5684c2524222e81 not found: ID does not exist"
Apr 22 20:12:51.424651 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:51.424538    2576 scope.go:117] "RemoveContainer" containerID="58013677f310d5ea4101737d28e4482a4874fa992a1bff716f614be3742999df"
Apr 22 20:12:51.424898 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:12:51.424869    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58013677f310d5ea4101737d28e4482a4874fa992a1bff716f614be3742999df\": container with ID starting with 58013677f310d5ea4101737d28e4482a4874fa992a1bff716f614be3742999df not found: ID does not exist" containerID="58013677f310d5ea4101737d28e4482a4874fa992a1bff716f614be3742999df"
Apr 22 20:12:51.424968 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:51.424905    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58013677f310d5ea4101737d28e4482a4874fa992a1bff716f614be3742999df"} err="failed to get container status \"58013677f310d5ea4101737d28e4482a4874fa992a1bff716f614be3742999df\": rpc error: code = NotFound desc = could not find container \"58013677f310d5ea4101737d28e4482a4874fa992a1bff716f614be3742999df\": container with ID starting with 58013677f310d5ea4101737d28e4482a4874fa992a1bff716f614be3742999df not found: ID does not exist"
Apr 22 20:12:53.375488 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:53.375443    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78754072-61b7-40b9-8931-323ca4258271" path="/var/lib/kubelet/pods/78754072-61b7-40b9-8931-323ca4258271/volumes"
Apr 22 20:12:56.546244 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:56.546203    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf" podUID="3c9eb4d1-7444-41d4-92e2-19873eda2d66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Apr 22 20:12:57.734860 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:57.734823    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:12:57.749403 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:57.749369    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:12:58.792485 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:58.792434    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"]
Apr 22 20:12:59.379908 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:12:59.379858    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl" podUID="2876a08a-0944-4f64-af5b-856c71d2f892" containerName="main" containerID="cri-o://21978615614c56e86be4fea00575811cce249a1b5b1e96f3dd5dbc9cf7f91075" gracePeriod=30
Apr 22 20:13:06.546224 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:06.546173    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf" podUID="3c9eb4d1-7444-41d4-92e2-19873eda2d66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Apr 22 20:13:16.547068 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:16.547017    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf" podUID="3c9eb4d1-7444-41d4-92e2-19873eda2d66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Apr 22 20:13:26.546372 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:26.546306    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf" podUID="3c9eb4d1-7444-41d4-92e2-19873eda2d66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Apr 22 20:13:27.313876 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.313840    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"]
Apr 22 20:13:27.314393 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.314373    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78754072-61b7-40b9-8931-323ca4258271" containerName="storage-initializer"
Apr 22 20:13:27.314502 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.314395    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="78754072-61b7-40b9-8931-323ca4258271" containerName="storage-initializer"
Apr 22 20:13:27.314502 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.314408    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78754072-61b7-40b9-8931-323ca4258271" containerName="main"
Apr 22 20:13:27.314502 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.314416    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="78754072-61b7-40b9-8931-323ca4258271" containerName="main"
Apr 22 20:13:27.314673 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.314520    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="78754072-61b7-40b9-8931-323ca4258271" containerName="main"
Apr 22 20:13:27.319246 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.319213    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:27.328727 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.328697    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"]
Apr 22 20:13:27.403418 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.403386    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-model-cache\") pod \"stop-feature-test-kserve-5bdb8cb8b8-fbpm6\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:27.403418 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.403422    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bddc7133-372c-4744-8fec-f7e6249e7fd9-tls-certs\") pod \"stop-feature-test-kserve-5bdb8cb8b8-fbpm6\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:27.403641 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.403452    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-dshm\") pod \"stop-feature-test-kserve-5bdb8cb8b8-fbpm6\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:27.403641 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.403477    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-kserve-provision-location\") pod \"stop-feature-test-kserve-5bdb8cb8b8-fbpm6\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:27.403641 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.403539    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6tc7\" (UniqueName: \"kubernetes.io/projected/bddc7133-372c-4744-8fec-f7e6249e7fd9-kube-api-access-n6tc7\") pod \"stop-feature-test-kserve-5bdb8cb8b8-fbpm6\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:27.403641 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.403566    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-home\") pod \"stop-feature-test-kserve-5bdb8cb8b8-fbpm6\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:27.504076 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.504043    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n6tc7\" (UniqueName: \"kubernetes.io/projected/bddc7133-372c-4744-8fec-f7e6249e7fd9-kube-api-access-n6tc7\") pod \"stop-feature-test-kserve-5bdb8cb8b8-fbpm6\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:27.504076 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.504081    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-home\") pod \"stop-feature-test-kserve-5bdb8cb8b8-fbpm6\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:27.504343 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.504187    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-model-cache\") pod \"stop-feature-test-kserve-5bdb8cb8b8-fbpm6\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:27.504343 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.504325    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bddc7133-372c-4744-8fec-f7e6249e7fd9-tls-certs\") pod \"stop-feature-test-kserve-5bdb8cb8b8-fbpm6\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:27.504455 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.504384    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-dshm\") pod \"stop-feature-test-kserve-5bdb8cb8b8-fbpm6\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:27.504455 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.504413    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-kserve-provision-location\") pod \"stop-feature-test-kserve-5bdb8cb8b8-fbpm6\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:27.504566 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.504539    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-model-cache\") pod \"stop-feature-test-kserve-5bdb8cb8b8-fbpm6\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:27.504622 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.504568    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-home\") pod \"stop-feature-test-kserve-5bdb8cb8b8-fbpm6\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:27.504931 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.504906    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-kserve-provision-location\") pod \"stop-feature-test-kserve-5bdb8cb8b8-fbpm6\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:27.506749 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.506730    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-dshm\") pod \"stop-feature-test-kserve-5bdb8cb8b8-fbpm6\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:27.507007 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.506989    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bddc7133-372c-4744-8fec-f7e6249e7fd9-tls-certs\") pod \"stop-feature-test-kserve-5bdb8cb8b8-fbpm6\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:27.512982 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.512960    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6tc7\" (UniqueName: \"kubernetes.io/projected/bddc7133-372c-4744-8fec-f7e6249e7fd9-kube-api-access-n6tc7\") pod \"stop-feature-test-kserve-5bdb8cb8b8-fbpm6\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:27.633482 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.633447    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:27.781685 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.781652    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"]
Apr 22 20:13:27.783944 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:13:27.783913    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbddc7133_372c_4744_8fec_f7e6249e7fd9.slice/crio-da86b6bf3d476b199be3be45d29429bb2b1a7c03360758a17400ea360587ff2b WatchSource:0}: Error finding container da86b6bf3d476b199be3be45d29429bb2b1a7c03360758a17400ea360587ff2b: Status 404 returned error can't find the container with id da86b6bf3d476b199be3be45d29429bb2b1a7c03360758a17400ea360587ff2b
Apr 22 20:13:27.785835 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:27.785817    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 22 20:13:28.504444 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:28.504399    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6" event={"ID":"bddc7133-372c-4744-8fec-f7e6249e7fd9","Type":"ContainerStarted","Data":"302aa5eb6489918a9c881b6c80b6a030768f1f012deac9a4e496b0c6063229c6"}
Apr 22 20:13:28.504710 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:28.504454    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6" event={"ID":"bddc7133-372c-4744-8fec-f7e6249e7fd9","Type":"ContainerStarted","Data":"da86b6bf3d476b199be3be45d29429bb2b1a7c03360758a17400ea360587ff2b"}
Apr 22 20:13:29.765893 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:29.765869    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5bdb8cb8b8-kglzl_2876a08a-0944-4f64-af5b-856c71d2f892/main/0.log"
Apr 22 20:13:29.766328 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:29.766313    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:13:29.932781 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:29.932695    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p77mj\" (UniqueName: \"kubernetes.io/projected/2876a08a-0944-4f64-af5b-856c71d2f892-kube-api-access-p77mj\") pod \"2876a08a-0944-4f64-af5b-856c71d2f892\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") "
Apr 22 20:13:29.932781 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:29.932743    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2876a08a-0944-4f64-af5b-856c71d2f892-tls-certs\") pod \"2876a08a-0944-4f64-af5b-856c71d2f892\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") "
Apr 22 20:13:29.932781 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:29.932781    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-model-cache\") pod \"2876a08a-0944-4f64-af5b-856c71d2f892\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") "
Apr 22 20:13:29.933056 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:29.932841    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-dshm\") pod \"2876a08a-0944-4f64-af5b-856c71d2f892\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") "
Apr 22 20:13:29.933056 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:29.932871    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-kserve-provision-location\") pod \"2876a08a-0944-4f64-af5b-856c71d2f892\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") "
Apr 22 20:13:29.933056 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:29.932895    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-home\") pod \"2876a08a-0944-4f64-af5b-856c71d2f892\" (UID: \"2876a08a-0944-4f64-af5b-856c71d2f892\") "
Apr 22 20:13:29.933056 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:29.933014    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-model-cache" (OuterVolumeSpecName: "model-cache") pod "2876a08a-0944-4f64-af5b-856c71d2f892" (UID: "2876a08a-0944-4f64-af5b-856c71d2f892"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:13:29.933334 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:29.933312    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-model-cache\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:13:29.933393 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:29.933351    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-home" (OuterVolumeSpecName: "home") pod "2876a08a-0944-4f64-af5b-856c71d2f892" (UID: "2876a08a-0944-4f64-af5b-856c71d2f892"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:13:29.935261 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:29.935234    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2876a08a-0944-4f64-af5b-856c71d2f892-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2876a08a-0944-4f64-af5b-856c71d2f892" (UID: "2876a08a-0944-4f64-af5b-856c71d2f892"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:13:29.935453 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:29.935434    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-dshm" (OuterVolumeSpecName: "dshm") pod "2876a08a-0944-4f64-af5b-856c71d2f892" (UID: "2876a08a-0944-4f64-af5b-856c71d2f892"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:13:29.935992 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:29.935970    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2876a08a-0944-4f64-af5b-856c71d2f892-kube-api-access-p77mj" (OuterVolumeSpecName: "kube-api-access-p77mj") pod "2876a08a-0944-4f64-af5b-856c71d2f892" (UID: "2876a08a-0944-4f64-af5b-856c71d2f892"). InnerVolumeSpecName "kube-api-access-p77mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:13:29.969082 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:29.969023    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2876a08a-0944-4f64-af5b-856c71d2f892" (UID: "2876a08a-0944-4f64-af5b-856c71d2f892"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:13:30.034760 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:30.034718    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-dshm\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:13:30.034760 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:30.034754    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-kserve-provision-location\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:13:30.034975 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:30.034769    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2876a08a-0944-4f64-af5b-856c71d2f892-home\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:13:30.034975 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:30.034781    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p77mj\" (UniqueName: \"kubernetes.io/projected/2876a08a-0944-4f64-af5b-856c71d2f892-kube-api-access-p77mj\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:13:30.034975 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:30.034791    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2876a08a-0944-4f64-af5b-856c71d2f892-tls-certs\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:13:30.515075 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:30.515045    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5bdb8cb8b8-kglzl_2876a08a-0944-4f64-af5b-856c71d2f892/main/0.log"
Apr 22 20:13:30.515524 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:30.515404    2576 generic.go:358] "Generic (PLEG): container finished" podID="2876a08a-0944-4f64-af5b-856c71d2f892" containerID="21978615614c56e86be4fea00575811cce249a1b5b1e96f3dd5dbc9cf7f91075" exitCode=137
Apr 22 20:13:30.515524 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:30.515502    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl" event={"ID":"2876a08a-0944-4f64-af5b-856c71d2f892","Type":"ContainerDied","Data":"21978615614c56e86be4fea00575811cce249a1b5b1e96f3dd5dbc9cf7f91075"}
Apr 22 20:13:30.515524 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:30.515513    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"
Apr 22 20:13:30.515770 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:30.515544    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl" event={"ID":"2876a08a-0944-4f64-af5b-856c71d2f892","Type":"ContainerDied","Data":"8a4114d1d4cf31bd2989b038d3c63d3e8cf7a55641681c3a5c79f9fc805a9238"}
Apr 22 20:13:30.515770 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:30.515564    2576 scope.go:117] "RemoveContainer" containerID="21978615614c56e86be4fea00575811cce249a1b5b1e96f3dd5dbc9cf7f91075"
Apr 22 20:13:30.538190 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:30.538163    2576 scope.go:117] "RemoveContainer" containerID="fca8dd54f96907cd8dd0530047ea7b80958dc011b76e1ebd8b298851923020ca"
Apr 22 20:13:30.547843 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:30.547798    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"]
Apr 22 20:13:30.552411 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:30.552377    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-kglzl"]
Apr 22 20:13:30.552984 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:30.552968    2576 scope.go:117] "RemoveContainer" containerID="21978615614c56e86be4fea00575811cce249a1b5b1e96f3dd5dbc9cf7f91075"
Apr 22 20:13:30.553410 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:13:30.553388    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21978615614c56e86be4fea00575811cce249a1b5b1e96f3dd5dbc9cf7f91075\": container with ID starting with 21978615614c56e86be4fea00575811cce249a1b5b1e96f3dd5dbc9cf7f91075 not found: ID does not exist" containerID="21978615614c56e86be4fea00575811cce249a1b5b1e96f3dd5dbc9cf7f91075"
Apr 22 20:13:30.553526 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:30.553419    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21978615614c56e86be4fea00575811cce249a1b5b1e96f3dd5dbc9cf7f91075"} err="failed to get container status \"21978615614c56e86be4fea00575811cce249a1b5b1e96f3dd5dbc9cf7f91075\": rpc error: code = NotFound desc = could not find container \"21978615614c56e86be4fea00575811cce249a1b5b1e96f3dd5dbc9cf7f91075\": container with ID starting with 21978615614c56e86be4fea00575811cce249a1b5b1e96f3dd5dbc9cf7f91075 not found: ID does not exist"
Apr 22 20:13:30.553526 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:30.553440    2576 scope.go:117] "RemoveContainer" containerID="fca8dd54f96907cd8dd0530047ea7b80958dc011b76e1ebd8b298851923020ca"
Apr 22 20:13:30.553742 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:13:30.553723    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca8dd54f96907cd8dd0530047ea7b80958dc011b76e1ebd8b298851923020ca\": container with ID starting with fca8dd54f96907cd8dd0530047ea7b80958dc011b76e1ebd8b298851923020ca not found: ID does not exist" containerID="fca8dd54f96907cd8dd0530047ea7b80958dc011b76e1ebd8b298851923020ca"
Apr 22 20:13:30.553808 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:30.553759    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca8dd54f96907cd8dd0530047ea7b80958dc011b76e1ebd8b298851923020ca"} err="failed to get container status \"fca8dd54f96907cd8dd0530047ea7b80958dc011b76e1ebd8b298851923020ca\": rpc error: code = NotFound desc = could not find container \"fca8dd54f96907cd8dd0530047ea7b80958dc011b76e1ebd8b298851923020ca\": container with ID starting with fca8dd54f96907cd8dd0530047ea7b80958dc011b76e1ebd8b298851923020ca not found: ID does not exist"
Apr 22 20:13:31.376584 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:31.376541    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2876a08a-0944-4f64-af5b-856c71d2f892" path="/var/lib/kubelet/pods/2876a08a-0944-4f64-af5b-856c71d2f892/volumes"
Apr 22 20:13:32.528455 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:32.528411    2576 generic.go:358] "Generic (PLEG): container finished" podID="bddc7133-372c-4744-8fec-f7e6249e7fd9" containerID="302aa5eb6489918a9c881b6c80b6a030768f1f012deac9a4e496b0c6063229c6" exitCode=0
Apr 22 20:13:32.528917 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:32.528486    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6" event={"ID":"bddc7133-372c-4744-8fec-f7e6249e7fd9","Type":"ContainerDied","Data":"302aa5eb6489918a9c881b6c80b6a030768f1f012deac9a4e496b0c6063229c6"}
Apr 22 20:13:33.535532 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:33.535495    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6" event={"ID":"bddc7133-372c-4744-8fec-f7e6249e7fd9","Type":"ContainerStarted","Data":"99c6174dbb29c0506968bc3bb15195c7e091416c832bcb6a2f8083c848be7ccf"}
Apr 22 20:13:33.556622 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:33.556556    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6" podStartSLOduration=6.556535257 podStartE2EDuration="6.556535257s" podCreationTimestamp="2026-04-22 20:13:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:13:33.554806159 +0000 UTC m=+948.654762424" watchObservedRunningTime="2026-04-22 20:13:33.556535257 +0000 UTC m=+948.656491524"
Apr 22 20:13:36.546443 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:36.546402    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf" podUID="3c9eb4d1-7444-41d4-92e2-19873eda2d66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Apr 22 20:13:37.634081 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:37.634034    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:37.634566 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:37.634096    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:13:37.635921 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:37.635854    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6" podUID="bddc7133-372c-4744-8fec-f7e6249e7fd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Apr 22 20:13:46.546554 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:46.546505    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf" podUID="3c9eb4d1-7444-41d4-92e2-19873eda2d66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Apr 22 20:13:47.634000 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:47.633955    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6" podUID="bddc7133-372c-4744-8fec-f7e6249e7fd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Apr 22 20:13:56.546535 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:56.546482    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf" podUID="3c9eb4d1-7444-41d4-92e2-19873eda2d66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Apr 22 20:13:57.634621 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:13:57.634570    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6" podUID="bddc7133-372c-4744-8fec-f7e6249e7fd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Apr 22 20:14:06.546821 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:06.546709    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf" podUID="3c9eb4d1-7444-41d4-92e2-19873eda2d66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Apr 22 20:14:07.634134 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:07.634081    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6" podUID="bddc7133-372c-4744-8fec-f7e6249e7fd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Apr 22 20:14:16.556447 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:16.556412    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:14:16.564510 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:16.564479    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:14:17.634477 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:17.634433    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6" podUID="bddc7133-372c-4744-8fec-f7e6249e7fd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Apr 22 20:14:22.742019 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:22.741976    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"]
Apr 22 20:14:22.742631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:22.742581    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf" podUID="3c9eb4d1-7444-41d4-92e2-19873eda2d66" containerName="main" containerID="cri-o://b7ced1d02c530a165bcbb5989034a6a569a3e19f9b8215d41e7721d55b5014cf" gracePeriod=30
Apr 22 20:14:27.634001 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:27.633960    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6" podUID="bddc7133-372c-4744-8fec-f7e6249e7fd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Apr 22 20:14:37.634168 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:37.634103    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6" podUID="bddc7133-372c-4744-8fec-f7e6249e7fd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Apr 22 20:14:45.395299 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.395264    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"]
Apr 22 20:14:45.395863 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.395844    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2876a08a-0944-4f64-af5b-856c71d2f892" containerName="storage-initializer"
Apr 22 20:14:45.395918 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.395868    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2876a08a-0944-4f64-af5b-856c71d2f892" containerName="storage-initializer"
Apr 22 20:14:45.395918 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.395907    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2876a08a-0944-4f64-af5b-856c71d2f892" containerName="main"
Apr 22 20:14:45.395988 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.395916    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2876a08a-0944-4f64-af5b-856c71d2f892" containerName="main"
Apr 22 20:14:45.396023 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.396015    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="2876a08a-0944-4f64-af5b-856c71d2f892" containerName="main"
Apr 22 20:14:45.406840 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.406704    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:45.410262 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.409738    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Apr 22 20:14:45.411804 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.411775    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"]
Apr 22 20:14:45.499383 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.499326    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-home\") pod \"router-with-refs-test-kserve-58bf87d84-c8qcm\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:45.499383 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.499387    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbbr9\" (UniqueName: \"kubernetes.io/projected/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-kube-api-access-lbbr9\") pod \"router-with-refs-test-kserve-58bf87d84-c8qcm\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:45.499658 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.499457    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-dshm\") pod \"router-with-refs-test-kserve-58bf87d84-c8qcm\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:45.499658 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.499486    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-tls-certs\") pod \"router-with-refs-test-kserve-58bf87d84-c8qcm\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:45.499658 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.499640    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-model-cache\") pod \"router-with-refs-test-kserve-58bf87d84-c8qcm\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:45.499806 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.499680    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-kserve-provision-location\") pod \"router-with-refs-test-kserve-58bf87d84-c8qcm\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:45.600935 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.600891    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-home\") pod \"router-with-refs-test-kserve-58bf87d84-c8qcm\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:45.601170 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.600976    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lbbr9\" (UniqueName: \"kubernetes.io/projected/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-kube-api-access-lbbr9\") pod \"router-with-refs-test-kserve-58bf87d84-c8qcm\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:45.601170 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.601025    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-dshm\") pod \"router-with-refs-test-kserve-58bf87d84-c8qcm\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:45.601170 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.601051    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-tls-certs\") pod \"router-with-refs-test-kserve-58bf87d84-c8qcm\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:45.601579 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.601389    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-home\") pod \"router-with-refs-test-kserve-58bf87d84-c8qcm\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:45.601727 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.601702    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-model-cache\") pod \"router-with-refs-test-kserve-58bf87d84-c8qcm\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:45.601954 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.601933    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-kserve-provision-location\") pod \"router-with-refs-test-kserve-58bf87d84-c8qcm\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:45.602278 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.602257    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-model-cache\") pod \"router-with-refs-test-kserve-58bf87d84-c8qcm\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:45.602441 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.602304    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-kserve-provision-location\") pod \"router-with-refs-test-kserve-58bf87d84-c8qcm\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:45.603844 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.603821    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-dshm\") pod \"router-with-refs-test-kserve-58bf87d84-c8qcm\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:45.603949 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.603860    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-tls-certs\") pod \"router-with-refs-test-kserve-58bf87d84-c8qcm\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:45.608996 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.608970    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbbr9\" (UniqueName: \"kubernetes.io/projected/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-kube-api-access-lbbr9\") pod \"router-with-refs-test-kserve-58bf87d84-c8qcm\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:45.723035 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.722940    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:45.867939 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.867895    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"]
Apr 22 20:14:45.870738 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:14:45.870701    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2564b4_c1b5_4ddc_85c0_3cced4e8eb6b.slice/crio-065a33753783227cfa43ec1ab2b328237db0f01b3e2bb6028fd1fa5f98fe1f02 WatchSource:0}: Error finding container 065a33753783227cfa43ec1ab2b328237db0f01b3e2bb6028fd1fa5f98fe1f02: Status 404 returned error can't find the container with id 065a33753783227cfa43ec1ab2b328237db0f01b3e2bb6028fd1fa5f98fe1f02
Apr 22 20:14:45.878325 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:45.878286    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm" event={"ID":"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b","Type":"ContainerStarted","Data":"065a33753783227cfa43ec1ab2b328237db0f01b3e2bb6028fd1fa5f98fe1f02"}
Apr 22 20:14:46.884317 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:46.884269    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm" event={"ID":"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b","Type":"ContainerStarted","Data":"337625ac003b9cb1bb1047ecbf6ac1bdaaacdd35ad9042865c52cae4990fae09"}
Apr 22 20:14:47.634743 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:47.634686    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6" podUID="bddc7133-372c-4744-8fec-f7e6249e7fd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Apr 22 20:14:50.904567 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:50.904519    2576 generic.go:358] "Generic (PLEG): container finished" podID="ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" containerID="337625ac003b9cb1bb1047ecbf6ac1bdaaacdd35ad9042865c52cae4990fae09" exitCode=0
Apr 22 20:14:50.904930 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:50.904597    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm" event={"ID":"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b","Type":"ContainerDied","Data":"337625ac003b9cb1bb1047ecbf6ac1bdaaacdd35ad9042865c52cae4990fae09"}
Apr 22 20:14:51.912685 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:51.912641    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm" event={"ID":"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b","Type":"ContainerStarted","Data":"e1bc82ffc39ca9e71f166c84480ee4d9d2698fd6da0ec4a8cb7a6abfd70e0410"}
Apr 22 20:14:51.932579 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:51.932517    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm" podStartSLOduration=6.932491006 podStartE2EDuration="6.932491006s" podCreationTimestamp="2026-04-22 20:14:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:14:51.930716661 +0000 UTC m=+1027.030672926" watchObservedRunningTime="2026-04-22 20:14:51.932491006 +0000 UTC m=+1027.032447270"
Apr 22 20:14:53.163551 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.163527    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-5c998f8fc6-rjndf_3c9eb4d1-7444-41d4-92e2-19873eda2d66/main/0.log"
Apr 22 20:14:53.163958 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.163939    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:14:53.279879 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.279803    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-model-cache\") pod \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") "
Apr 22 20:14:53.279879 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.279843    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-home\") pod \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") "
Apr 22 20:14:53.280103 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.279881    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9eb4d1-7444-41d4-92e2-19873eda2d66-tls-certs\") pod \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") "
Apr 22 20:14:53.280103 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.279905    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-dshm\") pod \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") "
Apr 22 20:14:53.280103 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.279944    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmlsg\" (UniqueName: \"kubernetes.io/projected/3c9eb4d1-7444-41d4-92e2-19873eda2d66-kube-api-access-vmlsg\") pod \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") "
Apr 22 20:14:53.280103 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.280055    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-kserve-provision-location\") pod \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\" (UID: \"3c9eb4d1-7444-41d4-92e2-19873eda2d66\") "
Apr 22 20:14:53.280347 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.280161    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c9eb4d1-7444-41d4-92e2-19873eda2d66" (UID: "3c9eb4d1-7444-41d4-92e2-19873eda2d66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:14:53.280408 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.280298    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-home" (OuterVolumeSpecName: "home") pod "3c9eb4d1-7444-41d4-92e2-19873eda2d66" (UID: "3c9eb4d1-7444-41d4-92e2-19873eda2d66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:14:53.280503 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.280482    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-model-cache\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:14:53.280503 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.280504    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-home\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:14:53.283002 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.282951    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c9eb4d1-7444-41d4-92e2-19873eda2d66-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c9eb4d1-7444-41d4-92e2-19873eda2d66" (UID: "3c9eb4d1-7444-41d4-92e2-19873eda2d66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:14:53.283002 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.282990    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9eb4d1-7444-41d4-92e2-19873eda2d66-kube-api-access-vmlsg" (OuterVolumeSpecName: "kube-api-access-vmlsg") pod "3c9eb4d1-7444-41d4-92e2-19873eda2d66" (UID: "3c9eb4d1-7444-41d4-92e2-19873eda2d66"). InnerVolumeSpecName "kube-api-access-vmlsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:14:53.283198 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.282996    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-dshm" (OuterVolumeSpecName: "dshm") pod "3c9eb4d1-7444-41d4-92e2-19873eda2d66" (UID: "3c9eb4d1-7444-41d4-92e2-19873eda2d66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:14:53.313903 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.313868    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3c9eb4d1-7444-41d4-92e2-19873eda2d66" (UID: "3c9eb4d1-7444-41d4-92e2-19873eda2d66"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:14:53.381244 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.381215    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9eb4d1-7444-41d4-92e2-19873eda2d66-tls-certs\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:14:53.381244 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.381239    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-dshm\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:14:53.381438 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.381249    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmlsg\" (UniqueName: \"kubernetes.io/projected/3c9eb4d1-7444-41d4-92e2-19873eda2d66-kube-api-access-vmlsg\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:14:53.381438 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.381268    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c9eb4d1-7444-41d4-92e2-19873eda2d66-kserve-provision-location\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:14:53.923024 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.922994    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-5c998f8fc6-rjndf_3c9eb4d1-7444-41d4-92e2-19873eda2d66/main/0.log"
Apr 22 20:14:53.923400 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.923377    2576 generic.go:358] "Generic (PLEG): container finished" podID="3c9eb4d1-7444-41d4-92e2-19873eda2d66" containerID="b7ced1d02c530a165bcbb5989034a6a569a3e19f9b8215d41e7721d55b5014cf" exitCode=137
Apr 22 20:14:53.923496 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.923464    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf" event={"ID":"3c9eb4d1-7444-41d4-92e2-19873eda2d66","Type":"ContainerDied","Data":"b7ced1d02c530a165bcbb5989034a6a569a3e19f9b8215d41e7721d55b5014cf"}
Apr 22 20:14:53.923537 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.923497    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf" event={"ID":"3c9eb4d1-7444-41d4-92e2-19873eda2d66","Type":"ContainerDied","Data":"4724dce6b93687bebab7f58d497bb9664d5142bcb9d4b81e1762032ee622d071"}
Apr 22 20:14:53.923537 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.923516    2576 scope.go:117] "RemoveContainer" containerID="b7ced1d02c530a165bcbb5989034a6a569a3e19f9b8215d41e7721d55b5014cf"
Apr 22 20:14:53.923601 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.923544    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"
Apr 22 20:14:53.942582 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.942540    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"]
Apr 22 20:14:53.951493 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.947656    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-5c998f8fc6-rjndf"]
Apr 22 20:14:53.952704 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:53.952682    2576 scope.go:117] "RemoveContainer" containerID="fc8b457938f49550a77b8f88637e54c1057cb634c5a69343a8a4cef6c6f5a33c"
Apr 22 20:14:54.005842 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:54.005815    2576 scope.go:117] "RemoveContainer" containerID="b7ced1d02c530a165bcbb5989034a6a569a3e19f9b8215d41e7721d55b5014cf"
Apr 22 20:14:54.006262 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:14:54.006231    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7ced1d02c530a165bcbb5989034a6a569a3e19f9b8215d41e7721d55b5014cf\": container with ID starting with b7ced1d02c530a165bcbb5989034a6a569a3e19f9b8215d41e7721d55b5014cf not found: ID does not exist" containerID="b7ced1d02c530a165bcbb5989034a6a569a3e19f9b8215d41e7721d55b5014cf"
Apr 22 20:14:54.006377 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:54.006277    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7ced1d02c530a165bcbb5989034a6a569a3e19f9b8215d41e7721d55b5014cf"} err="failed to get container status \"b7ced1d02c530a165bcbb5989034a6a569a3e19f9b8215d41e7721d55b5014cf\": rpc error: code = NotFound desc = could not find container \"b7ced1d02c530a165bcbb5989034a6a569a3e19f9b8215d41e7721d55b5014cf\": container with ID starting with b7ced1d02c530a165bcbb5989034a6a569a3e19f9b8215d41e7721d55b5014cf not found: ID does not exist"
Apr 22 20:14:54.006377 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:54.006307    2576 scope.go:117] "RemoveContainer" containerID="fc8b457938f49550a77b8f88637e54c1057cb634c5a69343a8a4cef6c6f5a33c"
Apr 22 20:14:54.006610 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:14:54.006588    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc8b457938f49550a77b8f88637e54c1057cb634c5a69343a8a4cef6c6f5a33c\": container with ID starting with fc8b457938f49550a77b8f88637e54c1057cb634c5a69343a8a4cef6c6f5a33c not found: ID does not exist" containerID="fc8b457938f49550a77b8f88637e54c1057cb634c5a69343a8a4cef6c6f5a33c"
Apr 22 20:14:54.006670 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:54.006620    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc8b457938f49550a77b8f88637e54c1057cb634c5a69343a8a4cef6c6f5a33c"} err="failed to get container status \"fc8b457938f49550a77b8f88637e54c1057cb634c5a69343a8a4cef6c6f5a33c\": rpc error: code = NotFound desc = could not find container \"fc8b457938f49550a77b8f88637e54c1057cb634c5a69343a8a4cef6c6f5a33c\": container with ID starting with fc8b457938f49550a77b8f88637e54c1057cb634c5a69343a8a4cef6c6f5a33c not found: ID does not exist"
Apr 22 20:14:55.379970 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:55.379929    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c9eb4d1-7444-41d4-92e2-19873eda2d66" path="/var/lib/kubelet/pods/3c9eb4d1-7444-41d4-92e2-19873eda2d66/volumes"
Apr 22 20:14:55.723499 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:55.723406    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:55.723499 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:55.723455    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:14:55.724684 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:55.724650    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm" podUID="ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Apr 22 20:14:57.634512 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:14:57.634475    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6" podUID="bddc7133-372c-4744-8fec-f7e6249e7fd9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Apr 22 20:15:05.723411 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:05.723366    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm" podUID="ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Apr 22 20:15:07.649897 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:07.649862    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:15:07.661938 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:07.661913    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:15:08.607047 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:08.607012    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"]
Apr 22 20:15:08.990091 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:08.990033    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6" podUID="bddc7133-372c-4744-8fec-f7e6249e7fd9" containerName="main" containerID="cri-o://99c6174dbb29c0506968bc3bb15195c7e091416c832bcb6a2f8083c848be7ccf" gracePeriod=30
Apr 22 20:15:15.723288 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:15.723248    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm" podUID="ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Apr 22 20:15:25.724200 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:25.724125    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm" podUID="ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Apr 22 20:15:35.723398 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:35.723312    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm" podUID="ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Apr 22 20:15:39.276366 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:39.276344    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5bdb8cb8b8-fbpm6_bddc7133-372c-4744-8fec-f7e6249e7fd9/main/0.log"
Apr 22 20:15:39.276741 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:39.276726    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:15:39.316120 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:39.316092    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-model-cache\") pod \"bddc7133-372c-4744-8fec-f7e6249e7fd9\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") "
Apr 22 20:15:39.316281 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:39.316136    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6tc7\" (UniqueName: \"kubernetes.io/projected/bddc7133-372c-4744-8fec-f7e6249e7fd9-kube-api-access-n6tc7\") pod \"bddc7133-372c-4744-8fec-f7e6249e7fd9\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") "
Apr 22 20:15:39.316281 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:39.316190    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-home\") pod \"bddc7133-372c-4744-8fec-f7e6249e7fd9\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") "
Apr 22 20:15:39.316281 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:39.316230    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bddc7133-372c-4744-8fec-f7e6249e7fd9-tls-certs\") pod \"bddc7133-372c-4744-8fec-f7e6249e7fd9\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") "
Apr 22 20:15:39.316451 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:39.316286    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-dshm\") pod \"bddc7133-372c-4744-8fec-f7e6249e7fd9\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") "
Apr 22 20:15:39.316451 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:39.316347    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-kserve-provision-location\") pod \"bddc7133-372c-4744-8fec-f7e6249e7fd9\" (UID: \"bddc7133-372c-4744-8fec-f7e6249e7fd9\") "
Apr 22 20:15:39.316451 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:39.316418    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-model-cache" (OuterVolumeSpecName: "model-cache") pod "bddc7133-372c-4744-8fec-f7e6249e7fd9" (UID: "bddc7133-372c-4744-8fec-f7e6249e7fd9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:15:39.316631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:39.316607    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-home" (OuterVolumeSpecName: "home") pod "bddc7133-372c-4744-8fec-f7e6249e7fd9" (UID: "bddc7133-372c-4744-8fec-f7e6249e7fd9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:15:39.316689 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:39.316673    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-model-cache\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:15:39.318520 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:39.318492    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bddc7133-372c-4744-8fec-f7e6249e7fd9-kube-api-access-n6tc7" (OuterVolumeSpecName: "kube-api-access-n6tc7") pod "bddc7133-372c-4744-8fec-f7e6249e7fd9" (UID: "bddc7133-372c-4744-8fec-f7e6249e7fd9"). InnerVolumeSpecName "kube-api-access-n6tc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:15:39.318670 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:39.318650    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bddc7133-372c-4744-8fec-f7e6249e7fd9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bddc7133-372c-4744-8fec-f7e6249e7fd9" (UID: "bddc7133-372c-4744-8fec-f7e6249e7fd9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:15:39.319033 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:39.319017    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-dshm" (OuterVolumeSpecName: "dshm") pod "bddc7133-372c-4744-8fec-f7e6249e7fd9" (UID: "bddc7133-372c-4744-8fec-f7e6249e7fd9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:15:39.362469 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:39.362425    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bddc7133-372c-4744-8fec-f7e6249e7fd9" (UID: "bddc7133-372c-4744-8fec-f7e6249e7fd9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:15:39.417108 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:39.417082    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6tc7\" (UniqueName: \"kubernetes.io/projected/bddc7133-372c-4744-8fec-f7e6249e7fd9-kube-api-access-n6tc7\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:15:39.417108 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:39.417107    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-home\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:15:39.417288 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:39.417116    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bddc7133-372c-4744-8fec-f7e6249e7fd9-tls-certs\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:15:39.417288 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:39.417125    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-dshm\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:15:39.417288 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:39.417133    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bddc7133-372c-4744-8fec-f7e6249e7fd9-kserve-provision-location\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:15:40.122651 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:40.122623    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5bdb8cb8b8-fbpm6_bddc7133-372c-4744-8fec-f7e6249e7fd9/main/0.log"
Apr 22 20:15:40.122970 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:40.122943    2576 generic.go:358] "Generic (PLEG): container finished" podID="bddc7133-372c-4744-8fec-f7e6249e7fd9" containerID="99c6174dbb29c0506968bc3bb15195c7e091416c832bcb6a2f8083c848be7ccf" exitCode=137
Apr 22 20:15:40.123037 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:40.123016    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"
Apr 22 20:15:40.123079 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:40.123043    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6" event={"ID":"bddc7133-372c-4744-8fec-f7e6249e7fd9","Type":"ContainerDied","Data":"99c6174dbb29c0506968bc3bb15195c7e091416c832bcb6a2f8083c848be7ccf"}
Apr 22 20:15:40.123114 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:40.123095    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6" event={"ID":"bddc7133-372c-4744-8fec-f7e6249e7fd9","Type":"ContainerDied","Data":"da86b6bf3d476b199be3be45d29429bb2b1a7c03360758a17400ea360587ff2b"}
Apr 22 20:15:40.123160 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:40.123121    2576 scope.go:117] "RemoveContainer" containerID="99c6174dbb29c0506968bc3bb15195c7e091416c832bcb6a2f8083c848be7ccf"
Apr 22 20:15:40.142909 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:40.142881    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"]
Apr 22 20:15:40.145380 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:40.145357    2576 scope.go:117] "RemoveContainer" containerID="302aa5eb6489918a9c881b6c80b6a030768f1f012deac9a4e496b0c6063229c6"
Apr 22 20:15:40.147403 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:40.147382    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5bdb8cb8b8-fbpm6"]
Apr 22 20:15:40.192480 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:40.192451    2576 scope.go:117] "RemoveContainer" containerID="99c6174dbb29c0506968bc3bb15195c7e091416c832bcb6a2f8083c848be7ccf"
Apr 22 20:15:40.192834 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:15:40.192803    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c6174dbb29c0506968bc3bb15195c7e091416c832bcb6a2f8083c848be7ccf\": container with ID starting with 99c6174dbb29c0506968bc3bb15195c7e091416c832bcb6a2f8083c848be7ccf not found: ID does not exist" containerID="99c6174dbb29c0506968bc3bb15195c7e091416c832bcb6a2f8083c848be7ccf"
Apr 22 20:15:40.192896 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:40.192846    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c6174dbb29c0506968bc3bb15195c7e091416c832bcb6a2f8083c848be7ccf"} err="failed to get container status \"99c6174dbb29c0506968bc3bb15195c7e091416c832bcb6a2f8083c848be7ccf\": rpc error: code = NotFound desc = could not find container \"99c6174dbb29c0506968bc3bb15195c7e091416c832bcb6a2f8083c848be7ccf\": container with ID starting with 99c6174dbb29c0506968bc3bb15195c7e091416c832bcb6a2f8083c848be7ccf not found: ID does not exist"
Apr 22 20:15:40.192896 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:40.192865    2576 scope.go:117] "RemoveContainer" containerID="302aa5eb6489918a9c881b6c80b6a030768f1f012deac9a4e496b0c6063229c6"
Apr 22 20:15:40.193189 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:15:40.193168    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"302aa5eb6489918a9c881b6c80b6a030768f1f012deac9a4e496b0c6063229c6\": container with ID starting with 302aa5eb6489918a9c881b6c80b6a030768f1f012deac9a4e496b0c6063229c6 not found: ID does not exist" containerID="302aa5eb6489918a9c881b6c80b6a030768f1f012deac9a4e496b0c6063229c6"
Apr 22 20:15:40.193238 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:40.193196    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"302aa5eb6489918a9c881b6c80b6a030768f1f012deac9a4e496b0c6063229c6"} err="failed to get container status \"302aa5eb6489918a9c881b6c80b6a030768f1f012deac9a4e496b0c6063229c6\": rpc error: code = NotFound desc = could not find container \"302aa5eb6489918a9c881b6c80b6a030768f1f012deac9a4e496b0c6063229c6\": container with ID starting with 302aa5eb6489918a9c881b6c80b6a030768f1f012deac9a4e496b0c6063229c6 not found: ID does not exist"
Apr 22 20:15:41.374477 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:41.374436    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bddc7133-372c-4744-8fec-f7e6249e7fd9" path="/var/lib/kubelet/pods/bddc7133-372c-4744-8fec-f7e6249e7fd9/volumes"
Apr 22 20:15:45.723317 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:45.723215    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm" podUID="ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Apr 22 20:15:55.723417 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:15:55.723370    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm" podUID="ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Apr 22 20:16:05.723324 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:05.723276    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm" podUID="ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Apr 22 20:16:15.723374 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:15.723328    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm" podUID="ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Apr 22 20:16:18.369492 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.369454    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"]
Apr 22 20:16:18.370256 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.370229    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bddc7133-372c-4744-8fec-f7e6249e7fd9" containerName="main"
Apr 22 20:16:18.370256 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.370256    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bddc7133-372c-4744-8fec-f7e6249e7fd9" containerName="main"
Apr 22 20:16:18.370256 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.370276    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9eb4d1-7444-41d4-92e2-19873eda2d66" containerName="storage-initializer"
Apr 22 20:16:18.370580 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.370285    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9eb4d1-7444-41d4-92e2-19873eda2d66" containerName="storage-initializer"
Apr 22 20:16:18.370580 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.370298    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9eb4d1-7444-41d4-92e2-19873eda2d66" containerName="main"
Apr 22 20:16:18.370580 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.370306    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9eb4d1-7444-41d4-92e2-19873eda2d66" containerName="main"
Apr 22 20:16:18.370580 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.370314    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bddc7133-372c-4744-8fec-f7e6249e7fd9" containerName="storage-initializer"
Apr 22 20:16:18.370580 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.370321    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bddc7133-372c-4744-8fec-f7e6249e7fd9" containerName="storage-initializer"
Apr 22 20:16:18.370580 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.370443    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="bddc7133-372c-4744-8fec-f7e6249e7fd9" containerName="main"
Apr 22 20:16:18.370580 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.370460    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="3c9eb4d1-7444-41d4-92e2-19873eda2d66" containerName="main"
Apr 22 20:16:18.375221 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.375197    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:18.377677 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.377654    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs\""
Apr 22 20:16:18.384588 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.384534    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"]
Apr 22 20:16:18.476264 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.476222    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54366896-2f71-4645-a622-0dbf65b25b52-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:18.476484 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.476299    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:18.476484 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.476423    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:18.476484 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.476451    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:18.476484 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.476477    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:18.476671 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.476524    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nldx\" (UniqueName: \"kubernetes.io/projected/54366896-2f71-4645-a622-0dbf65b25b52-kube-api-access-5nldx\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:18.578075 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.578032    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:18.578075 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.578074    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:18.578362 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.578209    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:18.578362 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.578257    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5nldx\" (UniqueName: \"kubernetes.io/projected/54366896-2f71-4645-a622-0dbf65b25b52-kube-api-access-5nldx\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:18.578362 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.578298    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54366896-2f71-4645-a622-0dbf65b25b52-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:18.578534 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.578458    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:18.578534 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.578497    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:18.578534 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.578507    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:18.578690 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.578572    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:18.580724 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.580701    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:18.581123 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.581102    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54366896-2f71-4645-a622-0dbf65b25b52-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:18.585880 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.585856    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nldx\" (UniqueName: \"kubernetes.io/projected/54366896-2f71-4645-a622-0dbf65b25b52-kube-api-access-5nldx\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:18.690262 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.690168    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:18.831331 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:18.831300    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"]
Apr 22 20:16:18.832583 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:16:18.832551    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54366896_2f71_4645_a622_0dbf65b25b52.slice/crio-3b014141d007359102bc9f1726ab87fa20de205c01207e8cd8204d5a731214d9 WatchSource:0}: Error finding container 3b014141d007359102bc9f1726ab87fa20de205c01207e8cd8204d5a731214d9: Status 404 returned error can't find the container with id 3b014141d007359102bc9f1726ab87fa20de205c01207e8cd8204d5a731214d9
Apr 22 20:16:19.298891 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:19.298055    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8" event={"ID":"54366896-2f71-4645-a622-0dbf65b25b52","Type":"ContainerStarted","Data":"6059358de7b2d6f2b227a69d81e3989f567d3c682aa2f56f263d4b3bdb4e1a94"}
Apr 22 20:16:19.298891 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:19.298101    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8" event={"ID":"54366896-2f71-4645-a622-0dbf65b25b52","Type":"ContainerStarted","Data":"3b014141d007359102bc9f1726ab87fa20de205c01207e8cd8204d5a731214d9"}
Apr 22 20:16:24.320587 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:24.320549    2576 generic.go:358] "Generic (PLEG): container finished" podID="54366896-2f71-4645-a622-0dbf65b25b52" containerID="6059358de7b2d6f2b227a69d81e3989f567d3c682aa2f56f263d4b3bdb4e1a94" exitCode=0
Apr 22 20:16:24.321053 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:24.320631    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8" event={"ID":"54366896-2f71-4645-a622-0dbf65b25b52","Type":"ContainerDied","Data":"6059358de7b2d6f2b227a69d81e3989f567d3c682aa2f56f263d4b3bdb4e1a94"}
Apr 22 20:16:25.326531 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:25.326497    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8" event={"ID":"54366896-2f71-4645-a622-0dbf65b25b52","Type":"ContainerStarted","Data":"6133b21694448804b474b69f45b9338c038cdcc7b3fe3580799fe5b91cf0f7b5"}
Apr 22 20:16:25.347665 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:25.347594    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8" podStartSLOduration=7.347574842 podStartE2EDuration="7.347574842s" podCreationTimestamp="2026-04-22 20:16:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:16:25.345076366 +0000 UTC m=+1120.445032653" watchObservedRunningTime="2026-04-22 20:16:25.347574842 +0000 UTC m=+1120.447531108"
Apr 22 20:16:25.733025 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:25.732992    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:16:25.740886 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:25.740854    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:16:28.691165 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:28.691114    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:28.691165 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:28.691167    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:16:28.692778 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:28.692745    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8" podUID="54366896-2f71-4645-a622-0dbf65b25b52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Apr 22 20:16:38.691076 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:38.691035    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8" podUID="54366896-2f71-4645-a622-0dbf65b25b52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Apr 22 20:16:39.898738 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:39.898690    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"]
Apr 22 20:16:39.899329 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:39.899206    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm" podUID="ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" containerName="main" containerID="cri-o://e1bc82ffc39ca9e71f166c84480ee4d9d2698fd6da0ec4a8cb7a6abfd70e0410" gracePeriod=30
Apr 22 20:16:48.691470 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:48.691419    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8" podUID="54366896-2f71-4645-a622-0dbf65b25b52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Apr 22 20:16:55.821248 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:55.821201    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"]
Apr 22 20:16:55.828678 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:55.828646    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:16:55.832266 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:55.832234    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
Apr 22 20:16:55.832639 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:55.832353    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-dockercfg-vgmkt\""
Apr 22 20:16:55.836822 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:55.836790    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"]
Apr 22 20:16:55.845332 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:55.845305    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"]
Apr 22 20:16:55.849094 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:55.849071    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:16:55.862687 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:55.862654    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"]
Apr 22 20:16:55.946722 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:55.946673    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:16:55.946899 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:55.946736    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a10ac615-0e39-4257-bed0-0fa0e77edabd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:16:55.946899 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:55.946793    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:16:55.946899 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:55.946871    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:16:55.947032 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:55.946910    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:16:55.947032 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:55.946950    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:16:55.947032 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:55.946976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:16:55.947032 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:55.947022    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thmt7\" (UniqueName: \"kubernetes.io/projected/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-kube-api-access-thmt7\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:16:55.947203 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:55.947051    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:16:55.947203 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:55.947181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h78t\" (UniqueName: \"kubernetes.io/projected/a10ac615-0e39-4257-bed0-0fa0e77edabd-kube-api-access-8h78t\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:16:55.947283 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:55.947232    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:16:55.947283 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:55.947268    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:16:56.047926 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.047873    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:16:56.048123 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.047942    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:16:56.048123 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.047983    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:16:56.048123 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.048011    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:16:56.048123 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.048043    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-thmt7\" (UniqueName: \"kubernetes.io/projected/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-kube-api-access-thmt7\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:16:56.048123 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.048073    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:16:56.048449 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.048125    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8h78t\" (UniqueName: \"kubernetes.io/projected/a10ac615-0e39-4257-bed0-0fa0e77edabd-kube-api-access-8h78t\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:16:56.048449 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.048194    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:16:56.048449 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.048224    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:16:56.048449 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.048263    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:16:56.048449 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.048291    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a10ac615-0e39-4257-bed0-0fa0e77edabd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:16:56.048449 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.048352    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:16:56.048449 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.048396    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:16:56.048992 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.048917    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:16:56.049109 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.049060    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:16:56.049205 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.049191    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:16:56.049272 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.049243    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:16:56.049686 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.049657    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:16:56.051191 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.051167    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:16:56.051295 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.051272    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:16:56.051392 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.051368    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:16:56.051851 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.051829    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a10ac615-0e39-4257-bed0-0fa0e77edabd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:16:56.056010 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.055985    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h78t\" (UniqueName: \"kubernetes.io/projected/a10ac615-0e39-4257-bed0-0fa0e77edabd-kube-api-access-8h78t\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:16:56.056010 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.055994    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thmt7\" (UniqueName: \"kubernetes.io/projected/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-kube-api-access-thmt7\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:16:56.146336 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.146275    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:16:56.163478 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.163440    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:16:56.318459 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.318429    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"]
Apr 22 20:16:56.319431 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:16:56.319404    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6422ae_8c66_4fe0_8b5d_f022557d8d60.slice/crio-decbf9f814f51babea8825b5454c4c301106a49789dc73b8dc00f35ec5c03c9d WatchSource:0}: Error finding container decbf9f814f51babea8825b5454c4c301106a49789dc73b8dc00f35ec5c03c9d: Status 404 returned error can't find the container with id decbf9f814f51babea8825b5454c4c301106a49789dc73b8dc00f35ec5c03c9d
Apr 22 20:16:56.340581 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.340554    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"]
Apr 22 20:16:56.343336 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:16:56.343302    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10ac615_0e39_4257_bed0_0fa0e77edabd.slice/crio-737cf6abf0bf646693587107e67f19a7d656288474c511e4b1457f687fce5e7d WatchSource:0}: Error finding container 737cf6abf0bf646693587107e67f19a7d656288474c511e4b1457f687fce5e7d: Status 404 returned error can't find the container with id 737cf6abf0bf646693587107e67f19a7d656288474c511e4b1457f687fce5e7d
Apr 22 20:16:56.467216 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.467171    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" event={"ID":"2b6422ae-8c66-4fe0-8b5d-f022557d8d60","Type":"ContainerStarted","Data":"decbf9f814f51babea8825b5454c4c301106a49789dc73b8dc00f35ec5c03c9d"}
Apr 22 20:16:56.469040 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.469005    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" event={"ID":"a10ac615-0e39-4257-bed0-0fa0e77edabd","Type":"ContainerStarted","Data":"174e022d76438ea2d3723b6f0d271ae56f7848c631809dad0278c8fb1ef65acc"}
Apr 22 20:16:56.469235 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:56.469048    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" event={"ID":"a10ac615-0e39-4257-bed0-0fa0e77edabd","Type":"ContainerStarted","Data":"737cf6abf0bf646693587107e67f19a7d656288474c511e4b1457f687fce5e7d"}
Apr 22 20:16:57.475573 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:57.475538    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" event={"ID":"2b6422ae-8c66-4fe0-8b5d-f022557d8d60","Type":"ContainerStarted","Data":"8e4b549f4c828a131a79f926ffdbe076a9024c2589d5f2856c59b9cdf6c7dc16"}
Apr 22 20:16:58.483041 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:58.483008    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" event={"ID":"2b6422ae-8c66-4fe0-8b5d-f022557d8d60","Type":"ContainerStarted","Data":"ab84ac86594b4ac918f19490f2b10f71fbecf4cfbcdc6d10a6c58a2605cc610a"}
Apr 22 20:16:58.483441 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:58.483311    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:16:58.691586 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:16:58.691528    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8" podUID="54366896-2f71-4645-a622-0dbf65b25b52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Apr 22 20:17:01.497811 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:01.497767    2576 generic.go:358] "Generic (PLEG): container finished" podID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerID="174e022d76438ea2d3723b6f0d271ae56f7848c631809dad0278c8fb1ef65acc" exitCode=0
Apr 22 20:17:01.498253 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:01.497843    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" event={"ID":"a10ac615-0e39-4257-bed0-0fa0e77edabd","Type":"ContainerDied","Data":"174e022d76438ea2d3723b6f0d271ae56f7848c631809dad0278c8fb1ef65acc"}
Apr 22 20:17:02.504044 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:02.503959    2576 generic.go:358] "Generic (PLEG): container finished" podID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerID="ab84ac86594b4ac918f19490f2b10f71fbecf4cfbcdc6d10a6c58a2605cc610a" exitCode=0
Apr 22 20:17:02.504499 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:02.504042    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" event={"ID":"2b6422ae-8c66-4fe0-8b5d-f022557d8d60","Type":"ContainerDied","Data":"ab84ac86594b4ac918f19490f2b10f71fbecf4cfbcdc6d10a6c58a2605cc610a"}
Apr 22 20:17:02.506204 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:02.506179    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" event={"ID":"a10ac615-0e39-4257-bed0-0fa0e77edabd","Type":"ContainerStarted","Data":"5cd001a55ed99c1c3f0262f7062ad8a35588c4527257ad3eb47c12d2d63eb4ce"}
Apr 22 20:17:02.547072 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:02.547012    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" podStartSLOduration=7.54699559 podStartE2EDuration="7.54699559s" podCreationTimestamp="2026-04-22 20:16:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:17:02.545453852 +0000 UTC m=+1157.645410118" watchObservedRunningTime="2026-04-22 20:17:02.54699559 +0000 UTC m=+1157.646951855"
Apr 22 20:17:03.514305 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:03.514267    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" event={"ID":"2b6422ae-8c66-4fe0-8b5d-f022557d8d60","Type":"ContainerStarted","Data":"9cce2e253ae83c701991ae89c1942bcfd9be6a7843cef192dfa68dcc2fb8ee6b"}
Apr 22 20:17:03.536538 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:03.536398    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" podStartSLOduration=7.613321989 podStartE2EDuration="8.536375006s" podCreationTimestamp="2026-04-22 20:16:55 +0000 UTC" firstStartedPulling="2026-04-22 20:16:56.321959233 +0000 UTC m=+1151.421915475" lastFinishedPulling="2026-04-22 20:16:57.245012246 +0000 UTC m=+1152.344968492" observedRunningTime="2026-04-22 20:17:03.534565117 +0000 UTC m=+1158.634521382" watchObservedRunningTime="2026-04-22 20:17:03.536375006 +0000 UTC m=+1158.636331281"
Apr 22 20:17:06.147112 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:06.147072    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:17:06.147602 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:06.147131    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:17:06.148522 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:06.148488    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Apr 22 20:17:06.164076 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:06.164043    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:17:06.164257 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:06.164093    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:17:06.165626 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:06.165591    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Apr 22 20:17:08.691110 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:08.691061    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8" podUID="54366896-2f71-4645-a622-0dbf65b25b52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Apr 22 20:17:10.259097 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.259069    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-58bf87d84-c8qcm_ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b/main/0.log"
Apr 22 20:17:10.260348 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.259518    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:17:10.301496 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.301452    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-kserve-provision-location\") pod \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") "
Apr 22 20:17:10.301692 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.301534    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbbr9\" (UniqueName: \"kubernetes.io/projected/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-kube-api-access-lbbr9\") pod \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") "
Apr 22 20:17:10.301692 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.301665    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-home\") pod \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") "
Apr 22 20:17:10.301819 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.301705    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-model-cache\") pod \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") "
Apr 22 20:17:10.301819 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.301731    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-dshm\") pod \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") "
Apr 22 20:17:10.301819 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.301807    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-tls-certs\") pod \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\" (UID: \"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b\") "
Apr 22 20:17:10.302122 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.302090    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" (UID: "ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:17:10.302433 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.302406    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-home" (OuterVolumeSpecName: "home") pod "ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" (UID: "ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:17:10.304982 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.304942    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-kube-api-access-lbbr9" (OuterVolumeSpecName: "kube-api-access-lbbr9") pod "ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" (UID: "ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b"). InnerVolumeSpecName "kube-api-access-lbbr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:17:10.304982 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.304959    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-dshm" (OuterVolumeSpecName: "dshm") pod "ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" (UID: "ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:17:10.305205 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.305182    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" (UID: "ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:17:10.384065 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.384010    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" (UID: "ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:17:10.402815 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.402778    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-home\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:17:10.402815 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.402809    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-model-cache\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:17:10.403070 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.402830    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-dshm\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:17:10.403070 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.402846    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-tls-certs\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:17:10.403070 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.402858    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-kserve-provision-location\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:17:10.403070 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.402871    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbbr9\" (UniqueName: \"kubernetes.io/projected/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b-kube-api-access-lbbr9\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:17:10.550734 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.550651    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-58bf87d84-c8qcm_ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b/main/0.log"
Apr 22 20:17:10.551105 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.551076    2576 generic.go:358] "Generic (PLEG): container finished" podID="ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" containerID="e1bc82ffc39ca9e71f166c84480ee4d9d2698fd6da0ec4a8cb7a6abfd70e0410" exitCode=137
Apr 22 20:17:10.551257 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.551187    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm" event={"ID":"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b","Type":"ContainerDied","Data":"e1bc82ffc39ca9e71f166c84480ee4d9d2698fd6da0ec4a8cb7a6abfd70e0410"}
Apr 22 20:17:10.551257 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.551233    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm" event={"ID":"ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b","Type":"ContainerDied","Data":"065a33753783227cfa43ec1ab2b328237db0f01b3e2bb6028fd1fa5f98fe1f02"}
Apr 22 20:17:10.551355 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.551256    2576 scope.go:117] "RemoveContainer" containerID="e1bc82ffc39ca9e71f166c84480ee4d9d2698fd6da0ec4a8cb7a6abfd70e0410"
Apr 22 20:17:10.551355 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.551202    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"
Apr 22 20:17:10.575934 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.575911    2576 scope.go:117] "RemoveContainer" containerID="337625ac003b9cb1bb1047ecbf6ac1bdaaacdd35ad9042865c52cae4990fae09"
Apr 22 20:17:10.580941 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.580908    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"]
Apr 22 20:17:10.585456 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.585427    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-58bf87d84-c8qcm"]
Apr 22 20:17:10.669779 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.669755    2576 scope.go:117] "RemoveContainer" containerID="e1bc82ffc39ca9e71f166c84480ee4d9d2698fd6da0ec4a8cb7a6abfd70e0410"
Apr 22 20:17:10.670202 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:17:10.670172    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1bc82ffc39ca9e71f166c84480ee4d9d2698fd6da0ec4a8cb7a6abfd70e0410\": container with ID starting with e1bc82ffc39ca9e71f166c84480ee4d9d2698fd6da0ec4a8cb7a6abfd70e0410 not found: ID does not exist" containerID="e1bc82ffc39ca9e71f166c84480ee4d9d2698fd6da0ec4a8cb7a6abfd70e0410"
Apr 22 20:17:10.670292 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.670224    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1bc82ffc39ca9e71f166c84480ee4d9d2698fd6da0ec4a8cb7a6abfd70e0410"} err="failed to get container status \"e1bc82ffc39ca9e71f166c84480ee4d9d2698fd6da0ec4a8cb7a6abfd70e0410\": rpc error: code = NotFound desc = could not find container \"e1bc82ffc39ca9e71f166c84480ee4d9d2698fd6da0ec4a8cb7a6abfd70e0410\": container with ID starting with e1bc82ffc39ca9e71f166c84480ee4d9d2698fd6da0ec4a8cb7a6abfd70e0410 not found: ID does not exist"
Apr 22 20:17:10.670292 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.670255    2576 scope.go:117] "RemoveContainer" containerID="337625ac003b9cb1bb1047ecbf6ac1bdaaacdd35ad9042865c52cae4990fae09"
Apr 22 20:17:10.670589 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:17:10.670559    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"337625ac003b9cb1bb1047ecbf6ac1bdaaacdd35ad9042865c52cae4990fae09\": container with ID starting with 337625ac003b9cb1bb1047ecbf6ac1bdaaacdd35ad9042865c52cae4990fae09 not found: ID does not exist" containerID="337625ac003b9cb1bb1047ecbf6ac1bdaaacdd35ad9042865c52cae4990fae09"
Apr 22 20:17:10.670702 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:10.670594    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337625ac003b9cb1bb1047ecbf6ac1bdaaacdd35ad9042865c52cae4990fae09"} err="failed to get container status \"337625ac003b9cb1bb1047ecbf6ac1bdaaacdd35ad9042865c52cae4990fae09\": rpc error: code = NotFound desc = could not find container \"337625ac003b9cb1bb1047ecbf6ac1bdaaacdd35ad9042865c52cae4990fae09\": container with ID starting with 337625ac003b9cb1bb1047ecbf6ac1bdaaacdd35ad9042865c52cae4990fae09 not found: ID does not exist"
Apr 22 20:17:11.382207 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:11.382140    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" path="/var/lib/kubelet/pods/ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b/volumes"
Apr 22 20:17:16.147299 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:16.147230    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Apr 22 20:17:16.160915 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:16.160882    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:17:16.164113 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:16.164077    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Apr 22 20:17:18.691292 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:18.691252    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8" podUID="54366896-2f71-4645-a622-0dbf65b25b52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Apr 22 20:17:26.147109 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:26.147058    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Apr 22 20:17:26.164542 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:26.164503    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Apr 22 20:17:28.691285 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:28.691230    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8" podUID="54366896-2f71-4645-a622-0dbf65b25b52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Apr 22 20:17:36.147808 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:36.147745    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Apr 22 20:17:36.164727 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:36.164681    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Apr 22 20:17:38.690837 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:38.690782    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8" podUID="54366896-2f71-4645-a622-0dbf65b25b52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Apr 22 20:17:46.148098 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:46.147674    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Apr 22 20:17:46.164725 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:46.164670    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Apr 22 20:17:48.691162 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:48.691094    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8" podUID="54366896-2f71-4645-a622-0dbf65b25b52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Apr 22 20:17:56.147315 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:56.147266    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Apr 22 20:17:56.164124 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:56.164085    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Apr 22 20:17:58.690949 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:17:58.690901    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8" podUID="54366896-2f71-4645-a622-0dbf65b25b52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Apr 22 20:18:06.146846 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:06.146796    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Apr 22 20:18:06.164772 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:06.164729    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Apr 22 20:18:08.701005 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:08.700955    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:18:08.708909 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:08.708879    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:18:16.147501 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:16.147439    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Apr 22 20:18:16.164535 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:16.164492    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Apr 22 20:18:20.320272 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:20.320239    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"]
Apr 22 20:18:20.320710 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:20.320587    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8" podUID="54366896-2f71-4645-a622-0dbf65b25b52" containerName="main" containerID="cri-o://6133b21694448804b474b69f45b9338c038cdcc7b3fe3580799fe5b91cf0f7b5" gracePeriod=30
Apr 22 20:18:26.147382 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:26.147334    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Apr 22 20:18:26.164033 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:26.163998    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Apr 22 20:18:26.881716 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:26.881680    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Apr 22 20:18:26.882116 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:26.882104    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" containerName="main"
Apr 22 20:18:26.882178 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:26.882119    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" containerName="main"
Apr 22 20:18:26.882178 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:26.882134    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" containerName="storage-initializer"
Apr 22 20:18:26.882178 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:26.882140    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" containerName="storage-initializer"
Apr 22 20:18:26.882287 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:26.882222    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="ce2564b4-c1b5-4ddc-85c0-3cced4e8eb6b" containerName="main"
Apr 22 20:18:26.886966 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:26.886941    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:26.889985 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:26.889959    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
Apr 22 20:18:26.890100 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:26.889964    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-ssqtz\""
Apr 22 20:18:26.895245 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:26.895217    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Apr 22 20:18:26.971729 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:26.971685    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:26.971729 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:26.971724    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:26.971978 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:26.971757    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28npt\" (UniqueName: \"kubernetes.io/projected/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-kube-api-access-28npt\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:26.971978 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:26.971871    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:26.971978 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:26.971922    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:26.971978 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:26.971949    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:27.073497 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:27.073456    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:27.073497 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:27.073491    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:27.073774 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:27.073512    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-28npt\" (UniqueName: \"kubernetes.io/projected/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-kube-api-access-28npt\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:27.073774 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:27.073557    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:27.073774 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:27.073600    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:27.073774 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:27.073628    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:27.073998 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:27.073925    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:27.073998 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:27.073972    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:27.074102 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:27.074024    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:27.075851 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:27.075829    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:27.076113 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:27.076095    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:27.080803 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:27.080786    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28npt\" (UniqueName: \"kubernetes.io/projected/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-kube-api-access-28npt\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:27.201484 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:27.201385    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:27.367721 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:27.367620    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Apr 22 20:18:27.369814 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:18:27.369780    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56b3a1c_c676_4d00_a1d4_754b8a3bb1fc.slice/crio-0b87e9fa1694ae69501912778e0657f27674bf5b8b562a251efbabdd65e25215 WatchSource:0}: Error finding container 0b87e9fa1694ae69501912778e0657f27674bf5b8b562a251efbabdd65e25215: Status 404 returned error can't find the container with id 0b87e9fa1694ae69501912778e0657f27674bf5b8b562a251efbabdd65e25215
Apr 22 20:18:27.920278 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:27.920245    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc","Type":"ContainerStarted","Data":"f5eac5c50043d0b7a5cee7dc71fcbbacba9d1f16503052d8fda0bb77d6cdc4c1"}
Apr 22 20:18:27.920451 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:27.920284    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc","Type":"ContainerStarted","Data":"0b87e9fa1694ae69501912778e0657f27674bf5b8b562a251efbabdd65e25215"}
Apr 22 20:18:31.943197 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:31.943159    2576 generic.go:358] "Generic (PLEG): container finished" podID="b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" containerID="f5eac5c50043d0b7a5cee7dc71fcbbacba9d1f16503052d8fda0bb77d6cdc4c1" exitCode=0
Apr 22 20:18:31.943566 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:31.943236    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc","Type":"ContainerDied","Data":"f5eac5c50043d0b7a5cee7dc71fcbbacba9d1f16503052d8fda0bb77d6cdc4c1"}
Apr 22 20:18:31.944525 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:31.944507    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 22 20:18:32.950516 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:32.950481    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc","Type":"ContainerStarted","Data":"5a0908a1bf1307a02834e819db7ddfabf3d9d5dccecc88d9322480ee5fa58762"}
Apr 22 20:18:32.968848 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:32.968786    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podStartSLOduration=6.968768701 podStartE2EDuration="6.968768701s" podCreationTimestamp="2026-04-22 20:18:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:18:32.967772068 +0000 UTC m=+1248.067728334" watchObservedRunningTime="2026-04-22 20:18:32.968768701 +0000 UTC m=+1248.068724956"
Apr 22 20:18:36.147658 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:36.147606    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Apr 22 20:18:36.164326 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:36.164284    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Apr 22 20:18:37.202573 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:37.202520    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:37.204213 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:37.204182    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Apr 22 20:18:46.147658 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:46.147610    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Apr 22 20:18:46.163879 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:46.163842    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Apr 22 20:18:47.201818 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:47.201771    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Apr 22 20:18:50.744919 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:50.744892    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8_54366896-2f71-4645-a622-0dbf65b25b52/main/0.log"
Apr 22 20:18:50.745384 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:50.745364    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:18:50.922256 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:50.922227    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nldx\" (UniqueName: \"kubernetes.io/projected/54366896-2f71-4645-a622-0dbf65b25b52-kube-api-access-5nldx\") pod \"54366896-2f71-4645-a622-0dbf65b25b52\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") "
Apr 22 20:18:50.922454 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:50.922271    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54366896-2f71-4645-a622-0dbf65b25b52-tls-certs\") pod \"54366896-2f71-4645-a622-0dbf65b25b52\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") "
Apr 22 20:18:50.922454 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:50.922330    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-kserve-provision-location\") pod \"54366896-2f71-4645-a622-0dbf65b25b52\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") "
Apr 22 20:18:50.922454 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:50.922410    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-home\") pod \"54366896-2f71-4645-a622-0dbf65b25b52\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") "
Apr 22 20:18:50.922454 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:50.922436    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-dshm\") pod \"54366896-2f71-4645-a622-0dbf65b25b52\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") "
Apr 22 20:18:50.922745 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:50.922534    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-model-cache\") pod \"54366896-2f71-4645-a622-0dbf65b25b52\" (UID: \"54366896-2f71-4645-a622-0dbf65b25b52\") "
Apr 22 20:18:50.924608 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:50.922909    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-home" (OuterVolumeSpecName: "home") pod "54366896-2f71-4645-a622-0dbf65b25b52" (UID: "54366896-2f71-4645-a622-0dbf65b25b52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:18:50.924608 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:50.923089    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-model-cache" (OuterVolumeSpecName: "model-cache") pod "54366896-2f71-4645-a622-0dbf65b25b52" (UID: "54366896-2f71-4645-a622-0dbf65b25b52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:18:50.924608 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:50.923356    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-model-cache\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:18:50.924608 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:50.923378    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-home\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:18:50.924968 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:50.924779    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54366896-2f71-4645-a622-0dbf65b25b52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54366896-2f71-4645-a622-0dbf65b25b52" (UID: "54366896-2f71-4645-a622-0dbf65b25b52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:18:50.925350 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:50.925326    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-dshm" (OuterVolumeSpecName: "dshm") pod "54366896-2f71-4645-a622-0dbf65b25b52" (UID: "54366896-2f71-4645-a622-0dbf65b25b52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:18:50.925561 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:50.925534    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54366896-2f71-4645-a622-0dbf65b25b52-kube-api-access-5nldx" (OuterVolumeSpecName: "kube-api-access-5nldx") pod "54366896-2f71-4645-a622-0dbf65b25b52" (UID: "54366896-2f71-4645-a622-0dbf65b25b52"). InnerVolumeSpecName "kube-api-access-5nldx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:18:50.962761 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:50.962710    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "54366896-2f71-4645-a622-0dbf65b25b52" (UID: "54366896-2f71-4645-a622-0dbf65b25b52"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:18:51.024981 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:51.024928    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-dshm\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:18:51.024981 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:51.024970    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nldx\" (UniqueName: \"kubernetes.io/projected/54366896-2f71-4645-a622-0dbf65b25b52-kube-api-access-5nldx\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:18:51.024981 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:51.024987    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54366896-2f71-4645-a622-0dbf65b25b52-tls-certs\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:18:51.025318 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:51.025004    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54366896-2f71-4645-a622-0dbf65b25b52-kserve-provision-location\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:18:51.042797 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:51.042771    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8_54366896-2f71-4645-a622-0dbf65b25b52/main/0.log"
Apr 22 20:18:51.043128 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:51.043106    2576 generic.go:358] "Generic (PLEG): container finished" podID="54366896-2f71-4645-a622-0dbf65b25b52" containerID="6133b21694448804b474b69f45b9338c038cdcc7b3fe3580799fe5b91cf0f7b5" exitCode=137
Apr 22 20:18:51.043272 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:51.043256    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"
Apr 22 20:18:51.043337 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:51.043273    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8" event={"ID":"54366896-2f71-4645-a622-0dbf65b25b52","Type":"ContainerDied","Data":"6133b21694448804b474b69f45b9338c038cdcc7b3fe3580799fe5b91cf0f7b5"}
Apr 22 20:18:51.043337 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:51.043329    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8" event={"ID":"54366896-2f71-4645-a622-0dbf65b25b52","Type":"ContainerDied","Data":"3b014141d007359102bc9f1726ab87fa20de205c01207e8cd8204d5a731214d9"}
Apr 22 20:18:51.043454 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:51.043351    2576 scope.go:117] "RemoveContainer" containerID="6133b21694448804b474b69f45b9338c038cdcc7b3fe3580799fe5b91cf0f7b5"
Apr 22 20:18:51.066514 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:51.065738    2576 scope.go:117] "RemoveContainer" containerID="6059358de7b2d6f2b227a69d81e3989f567d3c682aa2f56f263d4b3bdb4e1a94"
Apr 22 20:18:51.067966 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:51.067938    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"]
Apr 22 20:18:51.073868 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:51.073843    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788494645dwrzt8"]
Apr 22 20:18:51.095222 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:51.095180    2576 scope.go:117] "RemoveContainer" containerID="6133b21694448804b474b69f45b9338c038cdcc7b3fe3580799fe5b91cf0f7b5"
Apr 22 20:18:51.095543 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:18:51.095521    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6133b21694448804b474b69f45b9338c038cdcc7b3fe3580799fe5b91cf0f7b5\": container with ID starting with 6133b21694448804b474b69f45b9338c038cdcc7b3fe3580799fe5b91cf0f7b5 not found: ID does not exist" containerID="6133b21694448804b474b69f45b9338c038cdcc7b3fe3580799fe5b91cf0f7b5"
Apr 22 20:18:51.095633 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:51.095554    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6133b21694448804b474b69f45b9338c038cdcc7b3fe3580799fe5b91cf0f7b5"} err="failed to get container status \"6133b21694448804b474b69f45b9338c038cdcc7b3fe3580799fe5b91cf0f7b5\": rpc error: code = NotFound desc = could not find container \"6133b21694448804b474b69f45b9338c038cdcc7b3fe3580799fe5b91cf0f7b5\": container with ID starting with 6133b21694448804b474b69f45b9338c038cdcc7b3fe3580799fe5b91cf0f7b5 not found: ID does not exist"
Apr 22 20:18:51.095633 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:51.095580    2576 scope.go:117] "RemoveContainer" containerID="6059358de7b2d6f2b227a69d81e3989f567d3c682aa2f56f263d4b3bdb4e1a94"
Apr 22 20:18:51.095894 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:18:51.095873    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6059358de7b2d6f2b227a69d81e3989f567d3c682aa2f56f263d4b3bdb4e1a94\": container with ID starting with 6059358de7b2d6f2b227a69d81e3989f567d3c682aa2f56f263d4b3bdb4e1a94 not found: ID does not exist" containerID="6059358de7b2d6f2b227a69d81e3989f567d3c682aa2f56f263d4b3bdb4e1a94"
Apr 22 20:18:51.095952 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:51.095900    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6059358de7b2d6f2b227a69d81e3989f567d3c682aa2f56f263d4b3bdb4e1a94"} err="failed to get container status \"6059358de7b2d6f2b227a69d81e3989f567d3c682aa2f56f263d4b3bdb4e1a94\": rpc error: code = NotFound desc = could not find container \"6059358de7b2d6f2b227a69d81e3989f567d3c682aa2f56f263d4b3bdb4e1a94\": container with ID starting with 6059358de7b2d6f2b227a69d81e3989f567d3c682aa2f56f263d4b3bdb4e1a94 not found: ID does not exist"
Apr 22 20:18:51.379764 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:51.379729    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54366896-2f71-4645-a622-0dbf65b25b52" path="/var/lib/kubelet/pods/54366896-2f71-4645-a622-0dbf65b25b52/volumes"
Apr 22 20:18:56.147514 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:56.147471    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Apr 22 20:18:56.164361 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:56.164318    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Apr 22 20:18:57.202038 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:57.201986    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:18:57.202561 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:18:57.202334    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Apr 22 20:19:06.147490 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:19:06.147437    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Apr 22 20:19:06.164310 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:19:06.164267    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Apr 22 20:19:07.202006 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:19:07.201966    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Apr 22 20:19:16.147188 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:19:16.147120    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Apr 22 20:19:16.164794 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:19:16.164753    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Apr 22 20:19:17.202454 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:19:17.202416    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Apr 22 20:19:26.147442 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:19:26.147370    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Apr 22 20:19:26.164031 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:19:26.163984    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Apr 22 20:19:27.202134 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:19:27.202091    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Apr 22 20:19:36.157041 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:19:36.157002    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:19:36.174252 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:19:36.174223    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:19:36.177982 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:19:36.177958    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:19:36.189079 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:19:36.189051    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:19:37.202530 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:19:37.202487    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Apr 22 20:19:47.202743 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:19:47.202690    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Apr 22 20:19:54.270257 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:19:54.270219    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"]
Apr 22 20:19:54.271020 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:19:54.270989    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerName="main" containerID="cri-o://5cd001a55ed99c1c3f0262f7062ad8a35588c4527257ad3eb47c12d2d63eb4ce" gracePeriod=30
Apr 22 20:19:54.273238 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:19:54.273213    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"]
Apr 22 20:19:54.273609 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:19:54.273539    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="main" containerID="cri-o://9cce2e253ae83c701991ae89c1942bcfd9be6a7843cef192dfa68dcc2fb8ee6b" gracePeriod=30
Apr 22 20:19:57.201909 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:19:57.201864    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Apr 22 20:20:07.202331 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:07.202230    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Apr 22 20:20:17.211482 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:17.211447    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:20:17.219362 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:17.219335    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:20:24.273880 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:24.273817    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="llm-d-routing-sidecar" containerID="cri-o://8e4b549f4c828a131a79f926ffdbe076a9024c2589d5f2856c59b9cdf6c7dc16" gracePeriod=2
Apr 22 20:20:24.451133 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:24.451102    2576 generic.go:358] "Generic (PLEG): container finished" podID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerID="8e4b549f4c828a131a79f926ffdbe076a9024c2589d5f2856c59b9cdf6c7dc16" exitCode=0
Apr 22 20:20:24.451309 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:24.451186    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" event={"ID":"2b6422ae-8c66-4fe0-8b5d-f022557d8d60","Type":"ContainerDied","Data":"8e4b549f4c828a131a79f926ffdbe076a9024c2589d5f2856c59b9cdf6c7dc16"}
Apr 22 20:20:24.949895 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:24.949874    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl_2b6422ae-8c66-4fe0-8b5d-f022557d8d60/main/0.log"
Apr 22 20:20:24.950535 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:24.950514    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:20:24.953268 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:24.953251    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:20:25.034834 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.034804    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-dshm\") pod \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") "
Apr 22 20:20:25.034834 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.034837    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-home\") pod \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") "
Apr 22 20:20:25.035040 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.034858    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thmt7\" (UniqueName: \"kubernetes.io/projected/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-kube-api-access-thmt7\") pod \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") "
Apr 22 20:20:25.035040 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.034878    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-tls-certs\") pod \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") "
Apr 22 20:20:25.035040 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.034898    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-kserve-provision-location\") pod \"a10ac615-0e39-4257-bed0-0fa0e77edabd\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") "
Apr 22 20:20:25.035040 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.034932    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8h78t\" (UniqueName: \"kubernetes.io/projected/a10ac615-0e39-4257-bed0-0fa0e77edabd-kube-api-access-8h78t\") pod \"a10ac615-0e39-4257-bed0-0fa0e77edabd\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") "
Apr 22 20:20:25.035040 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.034947    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-model-cache\") pod \"a10ac615-0e39-4257-bed0-0fa0e77edabd\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") "
Apr 22 20:20:25.035040 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.034969    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-dshm\") pod \"a10ac615-0e39-4257-bed0-0fa0e77edabd\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") "
Apr 22 20:20:25.035040 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.034990    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-home\") pod \"a10ac615-0e39-4257-bed0-0fa0e77edabd\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") "
Apr 22 20:20:25.035040 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.035019    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a10ac615-0e39-4257-bed0-0fa0e77edabd-tls-certs\") pod \"a10ac615-0e39-4257-bed0-0fa0e77edabd\" (UID: \"a10ac615-0e39-4257-bed0-0fa0e77edabd\") "
Apr 22 20:20:25.035465 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.035057    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-model-cache\") pod \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") "
Apr 22 20:20:25.035465 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.035103    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-kserve-provision-location\") pod \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\" (UID: \"2b6422ae-8c66-4fe0-8b5d-f022557d8d60\") "
Apr 22 20:20:25.035465 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.035280    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-home" (OuterVolumeSpecName: "home") pod "2b6422ae-8c66-4fe0-8b5d-f022557d8d60" (UID: "2b6422ae-8c66-4fe0-8b5d-f022557d8d60"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:20:25.035788 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.035537    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-home\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:20:25.037328 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.037298    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-dshm" (OuterVolumeSpecName: "dshm") pod "2b6422ae-8c66-4fe0-8b5d-f022557d8d60" (UID: "2b6422ae-8c66-4fe0-8b5d-f022557d8d60"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:20:25.038064 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.037726    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-home" (OuterVolumeSpecName: "home") pod "a10ac615-0e39-4257-bed0-0fa0e77edabd" (UID: "a10ac615-0e39-4257-bed0-0fa0e77edabd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:20:25.038064 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.037818    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b6422ae-8c66-4fe0-8b5d-f022557d8d60" (UID: "2b6422ae-8c66-4fe0-8b5d-f022557d8d60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:20:25.038064 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.038042    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-model-cache" (OuterVolumeSpecName: "model-cache") pod "2b6422ae-8c66-4fe0-8b5d-f022557d8d60" (UID: "2b6422ae-8c66-4fe0-8b5d-f022557d8d60"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:20:25.038275 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.038057    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-model-cache" (OuterVolumeSpecName: "model-cache") pod "a10ac615-0e39-4257-bed0-0fa0e77edabd" (UID: "a10ac615-0e39-4257-bed0-0fa0e77edabd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:20:25.038275 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.038123    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-kube-api-access-thmt7" (OuterVolumeSpecName: "kube-api-access-thmt7") pod "2b6422ae-8c66-4fe0-8b5d-f022557d8d60" (UID: "2b6422ae-8c66-4fe0-8b5d-f022557d8d60"). InnerVolumeSpecName "kube-api-access-thmt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:20:25.038275 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.038167    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-dshm" (OuterVolumeSpecName: "dshm") pod "a10ac615-0e39-4257-bed0-0fa0e77edabd" (UID: "a10ac615-0e39-4257-bed0-0fa0e77edabd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:20:25.038275 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.038257    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10ac615-0e39-4257-bed0-0fa0e77edabd-kube-api-access-8h78t" (OuterVolumeSpecName: "kube-api-access-8h78t") pod "a10ac615-0e39-4257-bed0-0fa0e77edabd" (UID: "a10ac615-0e39-4257-bed0-0fa0e77edabd"). InnerVolumeSpecName "kube-api-access-8h78t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:20:25.038772 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.038752    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a10ac615-0e39-4257-bed0-0fa0e77edabd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a10ac615-0e39-4257-bed0-0fa0e77edabd" (UID: "a10ac615-0e39-4257-bed0-0fa0e77edabd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:20:25.100874 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.100807    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2b6422ae-8c66-4fe0-8b5d-f022557d8d60" (UID: "2b6422ae-8c66-4fe0-8b5d-f022557d8d60"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:20:25.104708 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.104680    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a10ac615-0e39-4257-bed0-0fa0e77edabd" (UID: "a10ac615-0e39-4257-bed0-0fa0e77edabd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:20:25.136302 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.136278    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h78t\" (UniqueName: \"kubernetes.io/projected/a10ac615-0e39-4257-bed0-0fa0e77edabd-kube-api-access-8h78t\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:20:25.136302 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.136300    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-model-cache\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:20:25.136447 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.136309    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-dshm\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:20:25.136447 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.136317    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-home\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:20:25.136447 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.136325    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a10ac615-0e39-4257-bed0-0fa0e77edabd-tls-certs\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:20:25.136447 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.136333    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-model-cache\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:20:25.136447 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.136340    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-kserve-provision-location\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:20:25.136447 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.136349    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-dshm\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:20:25.136447 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.136358    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thmt7\" (UniqueName: \"kubernetes.io/projected/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-kube-api-access-thmt7\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:20:25.136447 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.136367    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b6422ae-8c66-4fe0-8b5d-f022557d8d60-tls-certs\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:20:25.136447 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.136378    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a10ac615-0e39-4257-bed0-0fa0e77edabd-kserve-provision-location\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:20:25.456628 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.456603    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl_2b6422ae-8c66-4fe0-8b5d-f022557d8d60/main/0.log"
Apr 22 20:20:25.457206 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.457180    2576 generic.go:358] "Generic (PLEG): container finished" podID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerID="9cce2e253ae83c701991ae89c1942bcfd9be6a7843cef192dfa68dcc2fb8ee6b" exitCode=137
Apr 22 20:20:25.457293 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.457258    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"
Apr 22 20:20:25.457359 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.457261    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" event={"ID":"2b6422ae-8c66-4fe0-8b5d-f022557d8d60","Type":"ContainerDied","Data":"9cce2e253ae83c701991ae89c1942bcfd9be6a7843cef192dfa68dcc2fb8ee6b"}
Apr 22 20:20:25.457418 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.457371    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl" event={"ID":"2b6422ae-8c66-4fe0-8b5d-f022557d8d60","Type":"ContainerDied","Data":"decbf9f814f51babea8825b5454c4c301106a49789dc73b8dc00f35ec5c03c9d"}
Apr 22 20:20:25.457418 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.457392    2576 scope.go:117] "RemoveContainer" containerID="9cce2e253ae83c701991ae89c1942bcfd9be6a7843cef192dfa68dcc2fb8ee6b"
Apr 22 20:20:25.459068 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.459032    2576 generic.go:358] "Generic (PLEG): container finished" podID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerID="5cd001a55ed99c1c3f0262f7062ad8a35588c4527257ad3eb47c12d2d63eb4ce" exitCode=137
Apr 22 20:20:25.459253 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.459078    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" event={"ID":"a10ac615-0e39-4257-bed0-0fa0e77edabd","Type":"ContainerDied","Data":"5cd001a55ed99c1c3f0262f7062ad8a35588c4527257ad3eb47c12d2d63eb4ce"}
Apr 22 20:20:25.459253 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.459089    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"
Apr 22 20:20:25.459253 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.459103    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6" event={"ID":"a10ac615-0e39-4257-bed0-0fa0e77edabd","Type":"ContainerDied","Data":"737cf6abf0bf646693587107e67f19a7d656288474c511e4b1457f687fce5e7d"}
Apr 22 20:20:25.474965 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.474941    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"]
Apr 22 20:20:25.479416 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.479400    2576 scope.go:117] "RemoveContainer" containerID="ab84ac86594b4ac918f19490f2b10f71fbecf4cfbcdc6d10a6c58a2605cc610a"
Apr 22 20:20:25.479704 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.479681    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-94f88d545-wwswl"]
Apr 22 20:20:25.489375 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.489349    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"]
Apr 22 20:20:25.493911 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.493890    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-7d7dgs6"]
Apr 22 20:20:25.540003 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.539981    2576 scope.go:117] "RemoveContainer" containerID="8e4b549f4c828a131a79f926ffdbe076a9024c2589d5f2856c59b9cdf6c7dc16"
Apr 22 20:20:25.547878 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.547847    2576 scope.go:117] "RemoveContainer" containerID="9cce2e253ae83c701991ae89c1942bcfd9be6a7843cef192dfa68dcc2fb8ee6b"
Apr 22 20:20:25.548139 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:20:25.548119    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cce2e253ae83c701991ae89c1942bcfd9be6a7843cef192dfa68dcc2fb8ee6b\": container with ID starting with 9cce2e253ae83c701991ae89c1942bcfd9be6a7843cef192dfa68dcc2fb8ee6b not found: ID does not exist" containerID="9cce2e253ae83c701991ae89c1942bcfd9be6a7843cef192dfa68dcc2fb8ee6b"
Apr 22 20:20:25.548231 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.548165    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cce2e253ae83c701991ae89c1942bcfd9be6a7843cef192dfa68dcc2fb8ee6b"} err="failed to get container status \"9cce2e253ae83c701991ae89c1942bcfd9be6a7843cef192dfa68dcc2fb8ee6b\": rpc error: code = NotFound desc = could not find container \"9cce2e253ae83c701991ae89c1942bcfd9be6a7843cef192dfa68dcc2fb8ee6b\": container with ID starting with 9cce2e253ae83c701991ae89c1942bcfd9be6a7843cef192dfa68dcc2fb8ee6b not found: ID does not exist"
Apr 22 20:20:25.548231 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.548186    2576 scope.go:117] "RemoveContainer" containerID="ab84ac86594b4ac918f19490f2b10f71fbecf4cfbcdc6d10a6c58a2605cc610a"
Apr 22 20:20:25.548461 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:20:25.548443    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab84ac86594b4ac918f19490f2b10f71fbecf4cfbcdc6d10a6c58a2605cc610a\": container with ID starting with ab84ac86594b4ac918f19490f2b10f71fbecf4cfbcdc6d10a6c58a2605cc610a not found: ID does not exist" containerID="ab84ac86594b4ac918f19490f2b10f71fbecf4cfbcdc6d10a6c58a2605cc610a"
Apr 22 20:20:25.548509 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.548470    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab84ac86594b4ac918f19490f2b10f71fbecf4cfbcdc6d10a6c58a2605cc610a"} err="failed to get container status \"ab84ac86594b4ac918f19490f2b10f71fbecf4cfbcdc6d10a6c58a2605cc610a\": rpc error: code = NotFound desc = could not find container \"ab84ac86594b4ac918f19490f2b10f71fbecf4cfbcdc6d10a6c58a2605cc610a\": container with ID starting with ab84ac86594b4ac918f19490f2b10f71fbecf4cfbcdc6d10a6c58a2605cc610a not found: ID does not exist"
Apr 22 20:20:25.548509 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.548486    2576 scope.go:117] "RemoveContainer" containerID="8e4b549f4c828a131a79f926ffdbe076a9024c2589d5f2856c59b9cdf6c7dc16"
Apr 22 20:20:25.548718 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:20:25.548695    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e4b549f4c828a131a79f926ffdbe076a9024c2589d5f2856c59b9cdf6c7dc16\": container with ID starting with 8e4b549f4c828a131a79f926ffdbe076a9024c2589d5f2856c59b9cdf6c7dc16 not found: ID does not exist" containerID="8e4b549f4c828a131a79f926ffdbe076a9024c2589d5f2856c59b9cdf6c7dc16"
Apr 22 20:20:25.548804 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.548721    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e4b549f4c828a131a79f926ffdbe076a9024c2589d5f2856c59b9cdf6c7dc16"} err="failed to get container status \"8e4b549f4c828a131a79f926ffdbe076a9024c2589d5f2856c59b9cdf6c7dc16\": rpc error: code = NotFound desc = could not find container \"8e4b549f4c828a131a79f926ffdbe076a9024c2589d5f2856c59b9cdf6c7dc16\": container with ID starting with 8e4b549f4c828a131a79f926ffdbe076a9024c2589d5f2856c59b9cdf6c7dc16 not found: ID does not exist"
Apr 22 20:20:25.548804 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.548738    2576 scope.go:117] "RemoveContainer" containerID="5cd001a55ed99c1c3f0262f7062ad8a35588c4527257ad3eb47c12d2d63eb4ce"
Apr 22 20:20:25.567985 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.567963    2576 scope.go:117] "RemoveContainer" containerID="174e022d76438ea2d3723b6f0d271ae56f7848c631809dad0278c8fb1ef65acc"
Apr 22 20:20:25.629712 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.629692    2576 scope.go:117] "RemoveContainer" containerID="5cd001a55ed99c1c3f0262f7062ad8a35588c4527257ad3eb47c12d2d63eb4ce"
Apr 22 20:20:25.629998 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:20:25.629974    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd001a55ed99c1c3f0262f7062ad8a35588c4527257ad3eb47c12d2d63eb4ce\": container with ID starting with 5cd001a55ed99c1c3f0262f7062ad8a35588c4527257ad3eb47c12d2d63eb4ce not found: ID does not exist" containerID="5cd001a55ed99c1c3f0262f7062ad8a35588c4527257ad3eb47c12d2d63eb4ce"
Apr 22 20:20:25.630101 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.630004    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd001a55ed99c1c3f0262f7062ad8a35588c4527257ad3eb47c12d2d63eb4ce"} err="failed to get container status \"5cd001a55ed99c1c3f0262f7062ad8a35588c4527257ad3eb47c12d2d63eb4ce\": rpc error: code = NotFound desc = could not find container \"5cd001a55ed99c1c3f0262f7062ad8a35588c4527257ad3eb47c12d2d63eb4ce\": container with ID starting with 5cd001a55ed99c1c3f0262f7062ad8a35588c4527257ad3eb47c12d2d63eb4ce not found: ID does not exist"
Apr 22 20:20:25.630101 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.630030    2576 scope.go:117] "RemoveContainer" containerID="174e022d76438ea2d3723b6f0d271ae56f7848c631809dad0278c8fb1ef65acc"
Apr 22 20:20:25.630299 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:20:25.630283    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"174e022d76438ea2d3723b6f0d271ae56f7848c631809dad0278c8fb1ef65acc\": container with ID starting with 174e022d76438ea2d3723b6f0d271ae56f7848c631809dad0278c8fb1ef65acc not found: ID does not exist" containerID="174e022d76438ea2d3723b6f0d271ae56f7848c631809dad0278c8fb1ef65acc"
Apr 22 20:20:25.630338 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:25.630304    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174e022d76438ea2d3723b6f0d271ae56f7848c631809dad0278c8fb1ef65acc"} err="failed to get container status \"174e022d76438ea2d3723b6f0d271ae56f7848c631809dad0278c8fb1ef65acc\": rpc error: code = NotFound desc = could not find container \"174e022d76438ea2d3723b6f0d271ae56f7848c631809dad0278c8fb1ef65acc\": container with ID starting with 174e022d76438ea2d3723b6f0d271ae56f7848c631809dad0278c8fb1ef65acc not found: ID does not exist"
Apr 22 20:20:27.375298 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:27.375269    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" path="/var/lib/kubelet/pods/2b6422ae-8c66-4fe0-8b5d-f022557d8d60/volumes"
Apr 22 20:20:27.375893 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:27.375877    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" path="/var/lib/kubelet/pods/a10ac615-0e39-4257-bed0-0fa0e77edabd/volumes"
Apr 22 20:20:34.441178 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:34.441123    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Apr 22 20:20:34.441521 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:34.441500    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" containerName="main" containerID="cri-o://5a0908a1bf1307a02834e819db7ddfabf3d9d5dccecc88d9322480ee5fa58762" gracePeriod=30
Apr 22 20:20:35.292419 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.292395    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:20:35.341884 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.341847    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-dshm\") pod \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") "
Apr 22 20:20:35.342065 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.341894    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28npt\" (UniqueName: \"kubernetes.io/projected/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-kube-api-access-28npt\") pod \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") "
Apr 22 20:20:35.342065 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.341931    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-kserve-provision-location\") pod \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") "
Apr 22 20:20:35.342065 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.341982    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-home\") pod \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") "
Apr 22 20:20:35.342065 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.342026    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-model-cache\") pod \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") "
Apr 22 20:20:35.342332 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.342078    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-tls-certs\") pod \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\" (UID: \"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc\") "
Apr 22 20:20:35.342418 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.342378    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" (UID: "b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:20:35.342484 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.342418    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-home" (OuterVolumeSpecName: "home") pod "b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" (UID: "b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:20:35.344276 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.344241    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-kube-api-access-28npt" (OuterVolumeSpecName: "kube-api-access-28npt") pod "b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" (UID: "b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc"). InnerVolumeSpecName "kube-api-access-28npt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:20:35.344401 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.344297    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-dshm" (OuterVolumeSpecName: "dshm") pod "b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" (UID: "b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:20:35.344756 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.344729    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" (UID: "b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:20:35.399518 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.399484    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" (UID: "b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:20:35.443752 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.443720    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-dshm\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:20:35.443752 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.443749    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28npt\" (UniqueName: \"kubernetes.io/projected/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-kube-api-access-28npt\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:20:35.444127 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.443761    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-kserve-provision-location\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:20:35.444127 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.443770    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-home\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:20:35.444127 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.443779    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-model-cache\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:20:35.444127 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.443786    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc-tls-certs\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:20:35.507540 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.507505    2576 generic.go:358] "Generic (PLEG): container finished" podID="b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" containerID="5a0908a1bf1307a02834e819db7ddfabf3d9d5dccecc88d9322480ee5fa58762" exitCode=0
Apr 22 20:20:35.507675 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.507549    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc","Type":"ContainerDied","Data":"5a0908a1bf1307a02834e819db7ddfabf3d9d5dccecc88d9322480ee5fa58762"}
Apr 22 20:20:35.507675 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.507580    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 22 20:20:35.507675 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.507590    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc","Type":"ContainerDied","Data":"0b87e9fa1694ae69501912778e0657f27674bf5b8b562a251efbabdd65e25215"}
Apr 22 20:20:35.507675 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.507609    2576 scope.go:117] "RemoveContainer" containerID="5a0908a1bf1307a02834e819db7ddfabf3d9d5dccecc88d9322480ee5fa58762"
Apr 22 20:20:35.528458 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.528436    2576 scope.go:117] "RemoveContainer" containerID="f5eac5c50043d0b7a5cee7dc71fcbbacba9d1f16503052d8fda0bb77d6cdc4c1"
Apr 22 20:20:35.529970 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.529953    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Apr 22 20:20:35.532584 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.532559    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Apr 22 20:20:35.539194 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.539173    2576 scope.go:117] "RemoveContainer" containerID="5a0908a1bf1307a02834e819db7ddfabf3d9d5dccecc88d9322480ee5fa58762"
Apr 22 20:20:35.539459 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:20:35.539441    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a0908a1bf1307a02834e819db7ddfabf3d9d5dccecc88d9322480ee5fa58762\": container with ID starting with 5a0908a1bf1307a02834e819db7ddfabf3d9d5dccecc88d9322480ee5fa58762 not found: ID does not exist" containerID="5a0908a1bf1307a02834e819db7ddfabf3d9d5dccecc88d9322480ee5fa58762"
Apr 22 20:20:35.539536 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.539466    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0908a1bf1307a02834e819db7ddfabf3d9d5dccecc88d9322480ee5fa58762"} err="failed to get container status \"5a0908a1bf1307a02834e819db7ddfabf3d9d5dccecc88d9322480ee5fa58762\": rpc error: code = NotFound desc = could not find container \"5a0908a1bf1307a02834e819db7ddfabf3d9d5dccecc88d9322480ee5fa58762\": container with ID starting with 5a0908a1bf1307a02834e819db7ddfabf3d9d5dccecc88d9322480ee5fa58762 not found: ID does not exist"
Apr 22 20:20:35.539536 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.539485    2576 scope.go:117] "RemoveContainer" containerID="f5eac5c50043d0b7a5cee7dc71fcbbacba9d1f16503052d8fda0bb77d6cdc4c1"
Apr 22 20:20:35.539735 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:20:35.539721    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5eac5c50043d0b7a5cee7dc71fcbbacba9d1f16503052d8fda0bb77d6cdc4c1\": container with ID starting with f5eac5c50043d0b7a5cee7dc71fcbbacba9d1f16503052d8fda0bb77d6cdc4c1 not found: ID does not exist" containerID="f5eac5c50043d0b7a5cee7dc71fcbbacba9d1f16503052d8fda0bb77d6cdc4c1"
Apr 22 20:20:35.539790 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:35.539742    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5eac5c50043d0b7a5cee7dc71fcbbacba9d1f16503052d8fda0bb77d6cdc4c1"} err="failed to get container status \"f5eac5c50043d0b7a5cee7dc71fcbbacba9d1f16503052d8fda0bb77d6cdc4c1\": rpc error: code = NotFound desc = could not find container \"f5eac5c50043d0b7a5cee7dc71fcbbacba9d1f16503052d8fda0bb77d6cdc4c1\": container with ID starting with f5eac5c50043d0b7a5cee7dc71fcbbacba9d1f16503052d8fda0bb77d6cdc4c1 not found: ID does not exist"
Apr 22 20:20:37.374644 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:37.374612    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" path="/var/lib/kubelet/pods/b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc/volumes"
Apr 22 20:20:39.045023 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.044992    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"]
Apr 22 20:20:39.045461 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045445    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54366896-2f71-4645-a622-0dbf65b25b52" containerName="main"
Apr 22 20:20:39.045502 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045463    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="54366896-2f71-4645-a622-0dbf65b25b52" containerName="main"
Apr 22 20:20:39.045502 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045475    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerName="storage-initializer"
Apr 22 20:20:39.045502 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045481    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerName="storage-initializer"
Apr 22 20:20:39.045502 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045493    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="llm-d-routing-sidecar"
Apr 22 20:20:39.045502 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045501    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="llm-d-routing-sidecar"
Apr 22 20:20:39.045684 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045508    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="main"
Apr 22 20:20:39.045684 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045514    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="main"
Apr 22 20:20:39.045684 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045521    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" containerName="main"
Apr 22 20:20:39.045684 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045525    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" containerName="main"
Apr 22 20:20:39.045684 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045536    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="storage-initializer"
Apr 22 20:20:39.045684 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045541    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="storage-initializer"
Apr 22 20:20:39.045684 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045551    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54366896-2f71-4645-a622-0dbf65b25b52" containerName="storage-initializer"
Apr 22 20:20:39.045684 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045557    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="54366896-2f71-4645-a622-0dbf65b25b52" containerName="storage-initializer"
Apr 22 20:20:39.045684 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045563    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerName="main"
Apr 22 20:20:39.045684 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045569    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerName="main"
Apr 22 20:20:39.045684 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045616    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" containerName="storage-initializer"
Apr 22 20:20:39.045684 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045622    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" containerName="storage-initializer"
Apr 22 20:20:39.046018 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045715    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="a10ac615-0e39-4257-bed0-0fa0e77edabd" containerName="main"
Apr 22 20:20:39.046018 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045729    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="main"
Apr 22 20:20:39.046018 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045739    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="54366896-2f71-4645-a622-0dbf65b25b52" containerName="main"
Apr 22 20:20:39.046018 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045744    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="2b6422ae-8c66-4fe0-8b5d-f022557d8d60" containerName="llm-d-routing-sidecar"
Apr 22 20:20:39.046018 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.045751    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="b56b3a1c-c676-4d00-a1d4-754b8a3bb1fc" containerName="main"
Apr 22 20:20:39.049012 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.048991    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:39.051443 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.051415    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Apr 22 20:20:39.052043 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.052026    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tlcdw\""
Apr 22 20:20:39.059186 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.059134    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"]
Apr 22 20:20:39.175144 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.175110    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-dshm\") pod \"scheduler-inline-config-test-kserve-67fd5c8857-rgq99\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:39.175313 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.175187    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-67fd5c8857-rgq99\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:39.175313 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.175219    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvjhb\" (UniqueName: \"kubernetes.io/projected/3387e52d-d288-41b8-9cec-63eb6d3346f0-kube-api-access-rvjhb\") pod \"scheduler-inline-config-test-kserve-67fd5c8857-rgq99\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:39.175313 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.175239    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-home\") pod \"scheduler-inline-config-test-kserve-67fd5c8857-rgq99\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:39.175419 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.175313    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-model-cache\") pod \"scheduler-inline-config-test-kserve-67fd5c8857-rgq99\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:39.175419 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.175340    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3387e52d-d288-41b8-9cec-63eb6d3346f0-tls-certs\") pod \"scheduler-inline-config-test-kserve-67fd5c8857-rgq99\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:39.276216 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.276180    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3387e52d-d288-41b8-9cec-63eb6d3346f0-tls-certs\") pod \"scheduler-inline-config-test-kserve-67fd5c8857-rgq99\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:39.276396 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.276237    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-dshm\") pod \"scheduler-inline-config-test-kserve-67fd5c8857-rgq99\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:39.276396 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.276283    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-67fd5c8857-rgq99\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:39.276396 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.276308    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rvjhb\" (UniqueName: \"kubernetes.io/projected/3387e52d-d288-41b8-9cec-63eb6d3346f0-kube-api-access-rvjhb\") pod \"scheduler-inline-config-test-kserve-67fd5c8857-rgq99\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:39.276396 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.276329    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-home\") pod \"scheduler-inline-config-test-kserve-67fd5c8857-rgq99\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:39.276396 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.276361    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-model-cache\") pod \"scheduler-inline-config-test-kserve-67fd5c8857-rgq99\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:39.276737 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.276715    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-67fd5c8857-rgq99\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:39.276827 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.276753    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-model-cache\") pod \"scheduler-inline-config-test-kserve-67fd5c8857-rgq99\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:39.276827 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.276797    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-home\") pod \"scheduler-inline-config-test-kserve-67fd5c8857-rgq99\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:39.278770 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.278732    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-dshm\") pod \"scheduler-inline-config-test-kserve-67fd5c8857-rgq99\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:39.278886 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.278871    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3387e52d-d288-41b8-9cec-63eb6d3346f0-tls-certs\") pod \"scheduler-inline-config-test-kserve-67fd5c8857-rgq99\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:39.287733 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.287709    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvjhb\" (UniqueName: \"kubernetes.io/projected/3387e52d-d288-41b8-9cec-63eb6d3346f0-kube-api-access-rvjhb\") pod \"scheduler-inline-config-test-kserve-67fd5c8857-rgq99\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:39.302833 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.302762    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"]
Apr 22 20:20:39.312207 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.312177    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:39.315007 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.314988    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-epp-sa-dockercfg-52d4q\""
Apr 22 20:20:39.325537 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.325516    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"]
Apr 22 20:20:39.361771 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.361738    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:39.377010 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.376968    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:39.377010 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.377016    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:39.377284 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.377138    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:39.377284 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.377230    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:39.377284 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.377267    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d361114-b660-4aaa-9624-647ce8e984ec-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:39.377449 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.377305    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2gk5\" (UniqueName: \"kubernetes.io/projected/1d361114-b660-4aaa-9624-647ce8e984ec-kube-api-access-k2gk5\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:39.478932 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.478895    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:39.479109 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.478947    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:39.479109 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.479022    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:39.479109 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.479079    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:39.479323 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.479120    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d361114-b660-4aaa-9624-647ce8e984ec-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:39.479323 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.479196    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k2gk5\" (UniqueName: \"kubernetes.io/projected/1d361114-b660-4aaa-9624-647ce8e984ec-kube-api-access-k2gk5\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:39.479635 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.479468    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:39.479635 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.479468    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:39.479748 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.479657    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:39.479748 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.479676    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:39.482046 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.482021    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d361114-b660-4aaa-9624-647ce8e984ec-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:39.487355 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.487333    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2gk5\" (UniqueName: \"kubernetes.io/projected/1d361114-b660-4aaa-9624-647ce8e984ec-kube-api-access-k2gk5\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:39.493269 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.493247    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"]
Apr 22 20:20:39.494668 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:20:39.494644    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3387e52d_d288_41b8_9cec_63eb6d3346f0.slice/crio-01d928391ff249bc8baf1e91546315c8995205909538d503c35c96702c02ce55 WatchSource:0}: Error finding container 01d928391ff249bc8baf1e91546315c8995205909538d503c35c96702c02ce55: Status 404 returned error can't find the container with id 01d928391ff249bc8baf1e91546315c8995205909538d503c35c96702c02ce55
Apr 22 20:20:39.525466 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.525437    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99" event={"ID":"3387e52d-d288-41b8-9cec-63eb6d3346f0","Type":"ContainerStarted","Data":"01d928391ff249bc8baf1e91546315c8995205909538d503c35c96702c02ce55"}
Apr 22 20:20:39.622762 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.622711    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:39.991857 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:20:39.991824    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d361114_b660_4aaa_9624_647ce8e984ec.slice/crio-a5c9254e4ea2ad2b9026af1176db74ea496f9a523f9724521614c3c79437cd40 WatchSource:0}: Error finding container a5c9254e4ea2ad2b9026af1176db74ea496f9a523f9724521614c3c79437cd40: Status 404 returned error can't find the container with id a5c9254e4ea2ad2b9026af1176db74ea496f9a523f9724521614c3c79437cd40
Apr 22 20:20:39.992258 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:39.992232    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"]
Apr 22 20:20:40.532039 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:40.531999    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db" event={"ID":"1d361114-b660-4aaa-9624-647ce8e984ec","Type":"ContainerStarted","Data":"6a00d16a4e67de4181af7f5aa6955894b9778af4a04af6ddb6cf1445df4129a0"}
Apr 22 20:20:40.532039 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:40.532045    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db" event={"ID":"1d361114-b660-4aaa-9624-647ce8e984ec","Type":"ContainerStarted","Data":"a5c9254e4ea2ad2b9026af1176db74ea496f9a523f9724521614c3c79437cd40"}
Apr 22 20:20:40.533419 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:40.533397    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99" event={"ID":"3387e52d-d288-41b8-9cec-63eb6d3346f0","Type":"ContainerStarted","Data":"87b1d1976468a0d471a43d2bd95d292a84382e2279ff2e1853d4f5a21b310faa"}
Apr 22 20:20:41.538541 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:41.538504    2576 generic.go:358] "Generic (PLEG): container finished" podID="1d361114-b660-4aaa-9624-647ce8e984ec" containerID="6a00d16a4e67de4181af7f5aa6955894b9778af4a04af6ddb6cf1445df4129a0" exitCode=0
Apr 22 20:20:41.538957 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:41.538603    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db" event={"ID":"1d361114-b660-4aaa-9624-647ce8e984ec","Type":"ContainerDied","Data":"6a00d16a4e67de4181af7f5aa6955894b9778af4a04af6ddb6cf1445df4129a0"}
Apr 22 20:20:42.549756 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:42.549711    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db" event={"ID":"1d361114-b660-4aaa-9624-647ce8e984ec","Type":"ContainerStarted","Data":"ffce3c8b1209a6e8e4df59b92b29816a1065b781ba6e78f08c60ee9c8ed41b2b"}
Apr 22 20:20:42.549756 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:42.549757    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db" event={"ID":"1d361114-b660-4aaa-9624-647ce8e984ec","Type":"ContainerStarted","Data":"a9c0cc2c067af32ca1c3ef73a133ce7b7611fe93c2f87931a6171a2ee0383898"}
Apr 22 20:20:42.550648 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:42.549855    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:42.569263 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:42.569198    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db" podStartSLOduration=3.5691805260000002 podStartE2EDuration="3.569180526s" podCreationTimestamp="2026-04-22 20:20:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:20:42.567258235 +0000 UTC m=+1377.667214499" watchObservedRunningTime="2026-04-22 20:20:42.569180526 +0000 UTC m=+1377.669136803"
Apr 22 20:20:44.559826 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:44.559794    2576 generic.go:358] "Generic (PLEG): container finished" podID="3387e52d-d288-41b8-9cec-63eb6d3346f0" containerID="87b1d1976468a0d471a43d2bd95d292a84382e2279ff2e1853d4f5a21b310faa" exitCode=0
Apr 22 20:20:44.560204 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:44.559869    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99" event={"ID":"3387e52d-d288-41b8-9cec-63eb6d3346f0","Type":"ContainerDied","Data":"87b1d1976468a0d471a43d2bd95d292a84382e2279ff2e1853d4f5a21b310faa"}
Apr 22 20:20:46.570939 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:46.570903    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99" event={"ID":"3387e52d-d288-41b8-9cec-63eb6d3346f0","Type":"ContainerStarted","Data":"32c9375c5a85f8f1c799cba9290ee62b94534535606ac7a796169d8fd98e3022"}
Apr 22 20:20:46.589817 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:46.589759    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99" podStartSLOduration=6.454204757 podStartE2EDuration="7.589741737s" podCreationTimestamp="2026-04-22 20:20:39 +0000 UTC" firstStartedPulling="2026-04-22 20:20:44.561096883 +0000 UTC m=+1379.661053125" lastFinishedPulling="2026-04-22 20:20:45.696633863 +0000 UTC m=+1380.796590105" observedRunningTime="2026-04-22 20:20:46.586859885 +0000 UTC m=+1381.686816150" watchObservedRunningTime="2026-04-22 20:20:46.589741737 +0000 UTC m=+1381.689698003"
Apr 22 20:20:49.362652 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:49.362618    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:49.362652 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:49.362660    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:49.375797 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:49.375773    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:49.593111 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:49.593085    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:20:49.622833 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:49.622802    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:49.622976 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:49.622943    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:49.625524 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:49.625495    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:20:50.586868 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:20:50.586841    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:21:12.595822 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:12.595791    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:21:14.096560 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.096530    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"]
Apr 22 20:21:14.096933 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.096816    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99" podUID="3387e52d-d288-41b8-9cec-63eb6d3346f0" containerName="main" containerID="cri-o://32c9375c5a85f8f1c799cba9290ee62b94534535606ac7a796169d8fd98e3022" gracePeriod=30
Apr 22 20:21:14.103506 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.103477    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"]
Apr 22 20:21:14.103906 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.103874    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db" podUID="1d361114-b660-4aaa-9624-647ce8e984ec" containerName="main" containerID="cri-o://a9c0cc2c067af32ca1c3ef73a133ce7b7611fe93c2f87931a6171a2ee0383898" gracePeriod=30
Apr 22 20:21:14.104049 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.103912    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db" podUID="1d361114-b660-4aaa-9624-647ce8e984ec" containerName="tokenizer" containerID="cri-o://ffce3c8b1209a6e8e4df59b92b29816a1065b781ba6e78f08c60ee9c8ed41b2b" gracePeriod=30
Apr 22 20:21:14.354418 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.354356    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:21:14.414942 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.414906    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvjhb\" (UniqueName: \"kubernetes.io/projected/3387e52d-d288-41b8-9cec-63eb6d3346f0-kube-api-access-rvjhb\") pod \"3387e52d-d288-41b8-9cec-63eb6d3346f0\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") "
Apr 22 20:21:14.415137 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.414969    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-kserve-provision-location\") pod \"3387e52d-d288-41b8-9cec-63eb6d3346f0\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") "
Apr 22 20:21:14.415137 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.415061    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-home\") pod \"3387e52d-d288-41b8-9cec-63eb6d3346f0\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") "
Apr 22 20:21:14.415137 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.415095    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3387e52d-d288-41b8-9cec-63eb6d3346f0-tls-certs\") pod \"3387e52d-d288-41b8-9cec-63eb6d3346f0\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") "
Apr 22 20:21:14.415137 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.415129    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-model-cache\") pod \"3387e52d-d288-41b8-9cec-63eb6d3346f0\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") "
Apr 22 20:21:14.415364 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.415192    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-dshm\") pod \"3387e52d-d288-41b8-9cec-63eb6d3346f0\" (UID: \"3387e52d-d288-41b8-9cec-63eb6d3346f0\") "
Apr 22 20:21:14.415364 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.415334    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-home" (OuterVolumeSpecName: "home") pod "3387e52d-d288-41b8-9cec-63eb6d3346f0" (UID: "3387e52d-d288-41b8-9cec-63eb6d3346f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:21:14.415473 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.415417    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "3387e52d-d288-41b8-9cec-63eb6d3346f0" (UID: "3387e52d-d288-41b8-9cec-63eb6d3346f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:21:14.415730 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.415624    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-home\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:21:14.415859 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.415736    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-model-cache\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:21:14.417403 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.417379    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3387e52d-d288-41b8-9cec-63eb6d3346f0-kube-api-access-rvjhb" (OuterVolumeSpecName: "kube-api-access-rvjhb") pod "3387e52d-d288-41b8-9cec-63eb6d3346f0" (UID: "3387e52d-d288-41b8-9cec-63eb6d3346f0"). InnerVolumeSpecName "kube-api-access-rvjhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:21:14.417761 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.417710    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3387e52d-d288-41b8-9cec-63eb6d3346f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3387e52d-d288-41b8-9cec-63eb6d3346f0" (UID: "3387e52d-d288-41b8-9cec-63eb6d3346f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:21:14.417958 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.417931    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-dshm" (OuterVolumeSpecName: "dshm") pod "3387e52d-d288-41b8-9cec-63eb6d3346f0" (UID: "3387e52d-d288-41b8-9cec-63eb6d3346f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:21:14.485043 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.485001    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3387e52d-d288-41b8-9cec-63eb6d3346f0" (UID: "3387e52d-d288-41b8-9cec-63eb6d3346f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:21:14.516632 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.516589    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-kserve-provision-location\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:21:14.516632 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.516631    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3387e52d-d288-41b8-9cec-63eb6d3346f0-tls-certs\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:21:14.516844 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.516644    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3387e52d-d288-41b8-9cec-63eb6d3346f0-dshm\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:21:14.516844 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.516653    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvjhb\" (UniqueName: \"kubernetes.io/projected/3387e52d-d288-41b8-9cec-63eb6d3346f0-kube-api-access-rvjhb\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:21:14.686384 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.686345    2576 generic.go:358] "Generic (PLEG): container finished" podID="3387e52d-d288-41b8-9cec-63eb6d3346f0" containerID="32c9375c5a85f8f1c799cba9290ee62b94534535606ac7a796169d8fd98e3022" exitCode=0
Apr 22 20:21:14.686593 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.686424    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99" event={"ID":"3387e52d-d288-41b8-9cec-63eb6d3346f0","Type":"ContainerDied","Data":"32c9375c5a85f8f1c799cba9290ee62b94534535606ac7a796169d8fd98e3022"}
Apr 22 20:21:14.686593 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.686443    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"
Apr 22 20:21:14.686593 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.686474    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99" event={"ID":"3387e52d-d288-41b8-9cec-63eb6d3346f0","Type":"ContainerDied","Data":"01d928391ff249bc8baf1e91546315c8995205909538d503c35c96702c02ce55"}
Apr 22 20:21:14.686593 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.686494    2576 scope.go:117] "RemoveContainer" containerID="32c9375c5a85f8f1c799cba9290ee62b94534535606ac7a796169d8fd98e3022"
Apr 22 20:21:14.688730 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.688695    2576 generic.go:358] "Generic (PLEG): container finished" podID="1d361114-b660-4aaa-9624-647ce8e984ec" containerID="a9c0cc2c067af32ca1c3ef73a133ce7b7611fe93c2f87931a6171a2ee0383898" exitCode=0
Apr 22 20:21:14.688843 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.688744    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db" event={"ID":"1d361114-b660-4aaa-9624-647ce8e984ec","Type":"ContainerDied","Data":"a9c0cc2c067af32ca1c3ef73a133ce7b7611fe93c2f87931a6171a2ee0383898"}
Apr 22 20:21:14.696404 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.696376    2576 scope.go:117] "RemoveContainer" containerID="87b1d1976468a0d471a43d2bd95d292a84382e2279ff2e1853d4f5a21b310faa"
Apr 22 20:21:14.710374 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.710350    2576 scope.go:117] "RemoveContainer" containerID="32c9375c5a85f8f1c799cba9290ee62b94534535606ac7a796169d8fd98e3022"
Apr 22 20:21:14.710657 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.710637    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"]
Apr 22 20:21:14.710657 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:21:14.710637    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c9375c5a85f8f1c799cba9290ee62b94534535606ac7a796169d8fd98e3022\": container with ID starting with 32c9375c5a85f8f1c799cba9290ee62b94534535606ac7a796169d8fd98e3022 not found: ID does not exist" containerID="32c9375c5a85f8f1c799cba9290ee62b94534535606ac7a796169d8fd98e3022"
Apr 22 20:21:14.710833 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.710670    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c9375c5a85f8f1c799cba9290ee62b94534535606ac7a796169d8fd98e3022"} err="failed to get container status \"32c9375c5a85f8f1c799cba9290ee62b94534535606ac7a796169d8fd98e3022\": rpc error: code = NotFound desc = could not find container \"32c9375c5a85f8f1c799cba9290ee62b94534535606ac7a796169d8fd98e3022\": container with ID starting with 32c9375c5a85f8f1c799cba9290ee62b94534535606ac7a796169d8fd98e3022 not found: ID does not exist"
Apr 22 20:21:14.710833 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.710695    2576 scope.go:117] "RemoveContainer" containerID="87b1d1976468a0d471a43d2bd95d292a84382e2279ff2e1853d4f5a21b310faa"
Apr 22 20:21:14.710983 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:21:14.710958    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b1d1976468a0d471a43d2bd95d292a84382e2279ff2e1853d4f5a21b310faa\": container with ID starting with 87b1d1976468a0d471a43d2bd95d292a84382e2279ff2e1853d4f5a21b310faa not found: ID does not exist" containerID="87b1d1976468a0d471a43d2bd95d292a84382e2279ff2e1853d4f5a21b310faa"
Apr 22 20:21:14.711043 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.710998    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b1d1976468a0d471a43d2bd95d292a84382e2279ff2e1853d4f5a21b310faa"} err="failed to get container status \"87b1d1976468a0d471a43d2bd95d292a84382e2279ff2e1853d4f5a21b310faa\": rpc error: code = NotFound desc = could not find container \"87b1d1976468a0d471a43d2bd95d292a84382e2279ff2e1853d4f5a21b310faa\": container with ID starting with 87b1d1976468a0d471a43d2bd95d292a84382e2279ff2e1853d4f5a21b310faa not found: ID does not exist"
Apr 22 20:21:14.714621 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:14.714596    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-67fd5c8857-rgq99"]
Apr 22 20:21:15.359736 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.359712    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:21:15.374712 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.374684    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3387e52d-d288-41b8-9cec-63eb6d3346f0" path="/var/lib/kubelet/pods/3387e52d-d288-41b8-9cec-63eb6d3346f0/volumes"
Apr 22 20:21:15.426965 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.426932    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2gk5\" (UniqueName: \"kubernetes.io/projected/1d361114-b660-4aaa-9624-647ce8e984ec-kube-api-access-k2gk5\") pod \"1d361114-b660-4aaa-9624-647ce8e984ec\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") "
Apr 22 20:21:15.427129 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.426973    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-kserve-provision-location\") pod \"1d361114-b660-4aaa-9624-647ce8e984ec\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") "
Apr 22 20:21:15.427129 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.427016    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-tokenizer-uds\") pod \"1d361114-b660-4aaa-9624-647ce8e984ec\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") "
Apr 22 20:21:15.427129 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.427076    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d361114-b660-4aaa-9624-647ce8e984ec-tls-certs\") pod \"1d361114-b660-4aaa-9624-647ce8e984ec\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") "
Apr 22 20:21:15.427129 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.427120    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-tokenizer-tmp\") pod \"1d361114-b660-4aaa-9624-647ce8e984ec\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") "
Apr 22 20:21:15.427365 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.427173    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-tokenizer-cache\") pod \"1d361114-b660-4aaa-9624-647ce8e984ec\" (UID: \"1d361114-b660-4aaa-9624-647ce8e984ec\") "
Apr 22 20:21:15.427421 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.427394    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "1d361114-b660-4aaa-9624-647ce8e984ec" (UID: "1d361114-b660-4aaa-9624-647ce8e984ec"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:21:15.427531 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.427513    2576 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-tokenizer-uds\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:21:15.427594 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.427519    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "1d361114-b660-4aaa-9624-647ce8e984ec" (UID: "1d361114-b660-4aaa-9624-647ce8e984ec"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:21:15.427594 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.427550    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "1d361114-b660-4aaa-9624-647ce8e984ec" (UID: "1d361114-b660-4aaa-9624-647ce8e984ec"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:21:15.427864 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.427844    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1d361114-b660-4aaa-9624-647ce8e984ec" (UID: "1d361114-b660-4aaa-9624-647ce8e984ec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:21:15.429486 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.429461    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d361114-b660-4aaa-9624-647ce8e984ec-kube-api-access-k2gk5" (OuterVolumeSpecName: "kube-api-access-k2gk5") pod "1d361114-b660-4aaa-9624-647ce8e984ec" (UID: "1d361114-b660-4aaa-9624-647ce8e984ec"). InnerVolumeSpecName "kube-api-access-k2gk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:21:15.429610 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.429543    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d361114-b660-4aaa-9624-647ce8e984ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d361114-b660-4aaa-9624-647ce8e984ec" (UID: "1d361114-b660-4aaa-9624-647ce8e984ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:21:15.528111 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.528026    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2gk5\" (UniqueName: \"kubernetes.io/projected/1d361114-b660-4aaa-9624-647ce8e984ec-kube-api-access-k2gk5\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:21:15.528111 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.528054    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-kserve-provision-location\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:21:15.528111 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.528064    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d361114-b660-4aaa-9624-647ce8e984ec-tls-certs\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:21:15.528111 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.528074    2576 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-tokenizer-tmp\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:21:15.528111 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.528082    2576 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d361114-b660-4aaa-9624-647ce8e984ec-tokenizer-cache\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:21:15.693709 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.693673    2576 generic.go:358] "Generic (PLEG): container finished" podID="1d361114-b660-4aaa-9624-647ce8e984ec" containerID="ffce3c8b1209a6e8e4df59b92b29816a1065b781ba6e78f08c60ee9c8ed41b2b" exitCode=0
Apr 22 20:21:15.693882 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.693730    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db" event={"ID":"1d361114-b660-4aaa-9624-647ce8e984ec","Type":"ContainerDied","Data":"ffce3c8b1209a6e8e4df59b92b29816a1065b781ba6e78f08c60ee9c8ed41b2b"}
Apr 22 20:21:15.693882 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.693757    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"
Apr 22 20:21:15.693882 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.693768    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db" event={"ID":"1d361114-b660-4aaa-9624-647ce8e984ec","Type":"ContainerDied","Data":"a5c9254e4ea2ad2b9026af1176db74ea496f9a523f9724521614c3c79437cd40"}
Apr 22 20:21:15.693882 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.693789    2576 scope.go:117] "RemoveContainer" containerID="ffce3c8b1209a6e8e4df59b92b29816a1065b781ba6e78f08c60ee9c8ed41b2b"
Apr 22 20:21:15.704141 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.704121    2576 scope.go:117] "RemoveContainer" containerID="a9c0cc2c067af32ca1c3ef73a133ce7b7611fe93c2f87931a6171a2ee0383898"
Apr 22 20:21:15.712809 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.712788    2576 scope.go:117] "RemoveContainer" containerID="6a00d16a4e67de4181af7f5aa6955894b9778af4a04af6ddb6cf1445df4129a0"
Apr 22 20:21:15.717006 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.716979    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"]
Apr 22 20:21:15.721222 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.721194    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5c87d8j5db"]
Apr 22 20:21:15.722102 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.722086    2576 scope.go:117] "RemoveContainer" containerID="ffce3c8b1209a6e8e4df59b92b29816a1065b781ba6e78f08c60ee9c8ed41b2b"
Apr 22 20:21:15.722416 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:21:15.722393    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffce3c8b1209a6e8e4df59b92b29816a1065b781ba6e78f08c60ee9c8ed41b2b\": container with ID starting with ffce3c8b1209a6e8e4df59b92b29816a1065b781ba6e78f08c60ee9c8ed41b2b not found: ID does not exist" containerID="ffce3c8b1209a6e8e4df59b92b29816a1065b781ba6e78f08c60ee9c8ed41b2b"
Apr 22 20:21:15.722498 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.722430    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffce3c8b1209a6e8e4df59b92b29816a1065b781ba6e78f08c60ee9c8ed41b2b"} err="failed to get container status \"ffce3c8b1209a6e8e4df59b92b29816a1065b781ba6e78f08c60ee9c8ed41b2b\": rpc error: code = NotFound desc = could not find container \"ffce3c8b1209a6e8e4df59b92b29816a1065b781ba6e78f08c60ee9c8ed41b2b\": container with ID starting with ffce3c8b1209a6e8e4df59b92b29816a1065b781ba6e78f08c60ee9c8ed41b2b not found: ID does not exist"
Apr 22 20:21:15.722498 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.722457    2576 scope.go:117] "RemoveContainer" containerID="a9c0cc2c067af32ca1c3ef73a133ce7b7611fe93c2f87931a6171a2ee0383898"
Apr 22 20:21:15.722730 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:21:15.722709    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9c0cc2c067af32ca1c3ef73a133ce7b7611fe93c2f87931a6171a2ee0383898\": container with ID starting with a9c0cc2c067af32ca1c3ef73a133ce7b7611fe93c2f87931a6171a2ee0383898 not found: ID does not exist" containerID="a9c0cc2c067af32ca1c3ef73a133ce7b7611fe93c2f87931a6171a2ee0383898"
Apr 22 20:21:15.722830 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.722734    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c0cc2c067af32ca1c3ef73a133ce7b7611fe93c2f87931a6171a2ee0383898"} err="failed to get container status \"a9c0cc2c067af32ca1c3ef73a133ce7b7611fe93c2f87931a6171a2ee0383898\": rpc error: code = NotFound desc = could not find container \"a9c0cc2c067af32ca1c3ef73a133ce7b7611fe93c2f87931a6171a2ee0383898\": container with ID starting with a9c0cc2c067af32ca1c3ef73a133ce7b7611fe93c2f87931a6171a2ee0383898 not found: ID does not exist"
Apr 22 20:21:15.722830 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.722751    2576 scope.go:117] "RemoveContainer" containerID="6a00d16a4e67de4181af7f5aa6955894b9778af4a04af6ddb6cf1445df4129a0"
Apr 22 20:21:15.722983 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:21:15.722960    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a00d16a4e67de4181af7f5aa6955894b9778af4a04af6ddb6cf1445df4129a0\": container with ID starting with 6a00d16a4e67de4181af7f5aa6955894b9778af4a04af6ddb6cf1445df4129a0 not found: ID does not exist" containerID="6a00d16a4e67de4181af7f5aa6955894b9778af4a04af6ddb6cf1445df4129a0"
Apr 22 20:21:15.723029 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:15.722990    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a00d16a4e67de4181af7f5aa6955894b9778af4a04af6ddb6cf1445df4129a0"} err="failed to get container status \"6a00d16a4e67de4181af7f5aa6955894b9778af4a04af6ddb6cf1445df4129a0\": rpc error: code = NotFound desc = could not find container \"6a00d16a4e67de4181af7f5aa6955894b9778af4a04af6ddb6cf1445df4129a0\": container with ID starting with 6a00d16a4e67de4181af7f5aa6955894b9778af4a04af6ddb6cf1445df4129a0 not found: ID does not exist"
Apr 22 20:21:17.375299 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:21:17.375268    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d361114-b660-4aaa-9624-647ce8e984ec" path="/var/lib/kubelet/pods/1d361114-b660-4aaa-9624-647ce8e984ec/volumes"
Apr 22 20:22:49.957450 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:49.957405    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"]
Apr 22 20:22:49.959792 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:49.958044    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3387e52d-d288-41b8-9cec-63eb6d3346f0" containerName="storage-initializer"
Apr 22 20:22:49.959792 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:49.958062    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3387e52d-d288-41b8-9cec-63eb6d3346f0" containerName="storage-initializer"
Apr 22 20:22:49.959792 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:49.958090    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d361114-b660-4aaa-9624-647ce8e984ec" containerName="tokenizer"
Apr 22 20:22:49.959792 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:49.958099    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d361114-b660-4aaa-9624-647ce8e984ec" containerName="tokenizer"
Apr 22 20:22:49.959792 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:49.958122    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3387e52d-d288-41b8-9cec-63eb6d3346f0" containerName="main"
Apr 22 20:22:49.959792 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:49.958130    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3387e52d-d288-41b8-9cec-63eb6d3346f0" containerName="main"
Apr 22 20:22:49.959792 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:49.958144    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d361114-b660-4aaa-9624-647ce8e984ec" containerName="main"
Apr 22 20:22:49.959792 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:49.958171    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d361114-b660-4aaa-9624-647ce8e984ec" containerName="main"
Apr 22 20:22:49.959792 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:49.958180    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d361114-b660-4aaa-9624-647ce8e984ec" containerName="storage-initializer"
Apr 22 20:22:49.959792 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:49.958188    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d361114-b660-4aaa-9624-647ce8e984ec" containerName="storage-initializer"
Apr 22 20:22:49.959792 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:49.958277    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="1d361114-b660-4aaa-9624-647ce8e984ec" containerName="main"
Apr 22 20:22:49.959792 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:49.958289    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="1d361114-b660-4aaa-9624-647ce8e984ec" containerName="tokenizer"
Apr 22 20:22:49.959792 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:49.958305    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="3387e52d-d288-41b8-9cec-63eb6d3346f0" containerName="main"
Apr 22 20:22:49.961598 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:49.961567    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:49.965879 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:49.965856    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-2-openshift-default-dockercfg-lw9bk\""
Apr 22 20:22:49.980792 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:49.980766    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"]
Apr 22 20:22:50.059040 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.058993    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7119c6-821e-4a5e-8827-a4c148facad7-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.059243 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.059183    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc7119c6-821e-4a5e-8827-a4c148facad7-istio-token\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.059325 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.059243    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7119c6-821e-4a5e-8827-a4c148facad7-workload-socket\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.059325 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.059286    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7119c6-821e-4a5e-8827-a4c148facad7-istio-podinfo\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.059416 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.059337    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7119c6-821e-4a5e-8827-a4c148facad7-istio-envoy\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.059416 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.059395    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7119c6-821e-4a5e-8827-a4c148facad7-istio-data\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.059499 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.059416    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7119c6-821e-4a5e-8827-a4c148facad7-credential-socket\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.059499 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.059448    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7119c6-821e-4a5e-8827-a4c148facad7-workload-certs\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.059499 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.059471    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zzxx\" (UniqueName: \"kubernetes.io/projected/bc7119c6-821e-4a5e-8827-a4c148facad7-kube-api-access-2zzxx\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.160604 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.160562    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7119c6-821e-4a5e-8827-a4c148facad7-istio-envoy\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.160604 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.160606    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7119c6-821e-4a5e-8827-a4c148facad7-istio-data\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.160859 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.160626    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7119c6-821e-4a5e-8827-a4c148facad7-credential-socket\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.160859 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.160649    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7119c6-821e-4a5e-8827-a4c148facad7-workload-certs\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.160859 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.160671    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2zzxx\" (UniqueName: \"kubernetes.io/projected/bc7119c6-821e-4a5e-8827-a4c148facad7-kube-api-access-2zzxx\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.160859 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.160711    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7119c6-821e-4a5e-8827-a4c148facad7-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.160859 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.160791    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc7119c6-821e-4a5e-8827-a4c148facad7-istio-token\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.160859 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.160833    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7119c6-821e-4a5e-8827-a4c148facad7-workload-socket\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.161199 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.160871    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7119c6-821e-4a5e-8827-a4c148facad7-istio-podinfo\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.161199 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.161094    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7119c6-821e-4a5e-8827-a4c148facad7-workload-certs\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.161199 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.161175    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7119c6-821e-4a5e-8827-a4c148facad7-istio-data\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.161373 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.161218    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7119c6-821e-4a5e-8827-a4c148facad7-credential-socket\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.161434 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.161379    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7119c6-821e-4a5e-8827-a4c148facad7-workload-socket\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.161631 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.161604    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7119c6-821e-4a5e-8827-a4c148facad7-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.163527 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.163499    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7119c6-821e-4a5e-8827-a4c148facad7-istio-envoy\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.163619 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.163591    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7119c6-821e-4a5e-8827-a4c148facad7-istio-podinfo\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.167821 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.167800    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc7119c6-821e-4a5e-8827-a4c148facad7-istio-token\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.168046 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.168026    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zzxx\" (UniqueName: \"kubernetes.io/projected/bc7119c6-821e-4a5e-8827-a4c148facad7-kube-api-access-2zzxx\") pod \"router-gateway-2-openshift-default-6866b85949-lmpz4\" (UID: \"bc7119c6-821e-4a5e-8827-a4c148facad7\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.281103 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.281007    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:50.431652 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.431611    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"]
Apr 22 20:22:50.433138 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:22:50.433111    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7119c6_821e_4a5e_8827_a4c148facad7.slice/crio-c693f7d0c7da5647e11959b811bc006db4195d6f967ad4e0007ebe0f99e6365f WatchSource:0}: Error finding container c693f7d0c7da5647e11959b811bc006db4195d6f967ad4e0007ebe0f99e6365f: Status 404 returned error can't find the container with id c693f7d0c7da5647e11959b811bc006db4195d6f967ad4e0007ebe0f99e6365f
Apr 22 20:22:50.435321 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.435285    2576 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Apr 22 20:22:50.435410 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.435367    2576 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Apr 22 20:22:50.435410 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:50.435401    2576 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Apr 22 20:22:51.084730 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:51.084692    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4" event={"ID":"bc7119c6-821e-4a5e-8827-a4c148facad7","Type":"ContainerStarted","Data":"7f5bcbe3795f46372e32039e5bb32adf903173a1fde47e10135c38ba5503ecd2"}
Apr 22 20:22:51.085187 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:51.084739    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4" event={"ID":"bc7119c6-821e-4a5e-8827-a4c148facad7","Type":"ContainerStarted","Data":"c693f7d0c7da5647e11959b811bc006db4195d6f967ad4e0007ebe0f99e6365f"}
Apr 22 20:22:51.106591 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:51.106526    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4" podStartSLOduration=2.106504147 podStartE2EDuration="2.106504147s" podCreationTimestamp="2026-04-22 20:22:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:22:51.102442748 +0000 UTC m=+1506.202399047" watchObservedRunningTime="2026-04-22 20:22:51.106504147 +0000 UTC m=+1506.206460413"
Apr 22 20:22:51.281876 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:51.281839    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:51.288424 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:51.288393    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:52.091444 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:52.091415    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:22:52.167269 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:22:52.167232    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-6866b85949-lmpz4"
Apr 22 20:23:01.597705 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.597670    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"]
Apr 22 20:23:01.602355 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.602333    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:01.604647 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.604619    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tlcdw\""
Apr 22 20:23:01.604785 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.604687    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-q7gqh\""
Apr 22 20:23:01.605100 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.605081    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Apr 22 20:23:01.607782 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.607761    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"]
Apr 22 20:23:01.611987 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.611967    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:01.616389 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.616362    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"]
Apr 22 20:23:01.630317 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.630239    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"]
Apr 22 20:23:01.669606 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.669560    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4154cf08-4faa-45bc-91f2-7618bc433c9f-tls-certs\") pod \"router-with-refs-pd-test-kserve-54b886bf4d-vdz6d\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:01.669606 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.669606    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-home\") pod \"router-with-refs-pd-test-kserve-54b886bf4d-vdz6d\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:01.669833 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.669631    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:01.669833 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.669701    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:01.669833 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.669748    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4gfr\" (UniqueName: \"kubernetes.io/projected/48359f5e-dad7-4ffa-b745-92f426d6df30-kube-api-access-d4gfr\") pod \"router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:01.669833 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.669793    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-54b886bf4d-vdz6d\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:01.669992 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.669856    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-model-cache\") pod \"router-with-refs-pd-test-kserve-54b886bf4d-vdz6d\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:01.669992 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.669889    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-home\") pod \"router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:01.669992 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.669905    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-dshm\") pod \"router-with-refs-pd-test-kserve-54b886bf4d-vdz6d\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:01.669992 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.669970    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:01.669992 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.669989    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48359f5e-dad7-4ffa-b745-92f426d6df30-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:01.670190 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.670014    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gndz\" (UniqueName: \"kubernetes.io/projected/4154cf08-4faa-45bc-91f2-7618bc433c9f-kube-api-access-9gndz\") pod \"router-with-refs-pd-test-kserve-54b886bf4d-vdz6d\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:01.771285 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.771249    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-dshm\") pod \"router-with-refs-pd-test-kserve-54b886bf4d-vdz6d\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:01.771474 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.771303    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:01.771474 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.771321    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48359f5e-dad7-4ffa-b745-92f426d6df30-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:01.771474 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.771344    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9gndz\" (UniqueName: \"kubernetes.io/projected/4154cf08-4faa-45bc-91f2-7618bc433c9f-kube-api-access-9gndz\") pod \"router-with-refs-pd-test-kserve-54b886bf4d-vdz6d\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:01.771474 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.771371    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4154cf08-4faa-45bc-91f2-7618bc433c9f-tls-certs\") pod \"router-with-refs-pd-test-kserve-54b886bf4d-vdz6d\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:01.771474 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.771388    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-home\") pod \"router-with-refs-pd-test-kserve-54b886bf4d-vdz6d\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:01.771751 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.771547    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:01.771751 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.771600    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:01.771751 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.771637    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d4gfr\" (UniqueName: \"kubernetes.io/projected/48359f5e-dad7-4ffa-b745-92f426d6df30-kube-api-access-d4gfr\") pod \"router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:01.771751 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.771686    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-54b886bf4d-vdz6d\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:01.771945 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.771778    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-model-cache\") pod \"router-with-refs-pd-test-kserve-54b886bf4d-vdz6d\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:01.771945 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.771815    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-home\") pod \"router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:01.771945 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.771825    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-home\") pod \"router-with-refs-pd-test-kserve-54b886bf4d-vdz6d\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:01.772104 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.771976    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:01.772262 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.772214    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-home\") pod \"router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:01.772354 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.772255    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:01.772538 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.772412    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-54b886bf4d-vdz6d\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:01.772658 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.772622    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-model-cache\") pod \"router-with-refs-pd-test-kserve-54b886bf4d-vdz6d\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:01.774447 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.774418    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:01.774551 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.774492    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-dshm\") pod \"router-with-refs-pd-test-kserve-54b886bf4d-vdz6d\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:01.774666 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.774643    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4154cf08-4faa-45bc-91f2-7618bc433c9f-tls-certs\") pod \"router-with-refs-pd-test-kserve-54b886bf4d-vdz6d\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:01.774705 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.774686    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48359f5e-dad7-4ffa-b745-92f426d6df30-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:01.778865 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.778836    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4gfr\" (UniqueName: \"kubernetes.io/projected/48359f5e-dad7-4ffa-b745-92f426d6df30-kube-api-access-d4gfr\") pod \"router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:01.779224 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.779205    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gndz\" (UniqueName: \"kubernetes.io/projected/4154cf08-4faa-45bc-91f2-7618bc433c9f-kube-api-access-9gndz\") pod \"router-with-refs-pd-test-kserve-54b886bf4d-vdz6d\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:01.916619 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.916576    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:01.946302 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:01.946265    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:02.069602 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:02.069544    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"]
Apr 22 20:23:02.073205 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:23:02.073142    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4154cf08_4faa_45bc_91f2_7618bc433c9f.slice/crio-8550af827ea6170856d922616c3a76a4291615298469e7a83d473cf3c22274fc WatchSource:0}: Error finding container 8550af827ea6170856d922616c3a76a4291615298469e7a83d473cf3c22274fc: Status 404 returned error can't find the container with id 8550af827ea6170856d922616c3a76a4291615298469e7a83d473cf3c22274fc
Apr 22 20:23:02.091565 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:02.091542    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"]
Apr 22 20:23:02.093299 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:23:02.093268    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48359f5e_dad7_4ffa_b745_92f426d6df30.slice/crio-fa356dc228f24936054aed23ee1feccea795b24a11872a0fedfc6ddd654756d4 WatchSource:0}: Error finding container fa356dc228f24936054aed23ee1feccea795b24a11872a0fedfc6ddd654756d4: Status 404 returned error can't find the container with id fa356dc228f24936054aed23ee1feccea795b24a11872a0fedfc6ddd654756d4
Apr 22 20:23:02.134501 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:02.134431    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf" event={"ID":"48359f5e-dad7-4ffa-b745-92f426d6df30","Type":"ContainerStarted","Data":"fa356dc228f24936054aed23ee1feccea795b24a11872a0fedfc6ddd654756d4"}
Apr 22 20:23:02.136090 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:02.136064    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" event={"ID":"4154cf08-4faa-45bc-91f2-7618bc433c9f","Type":"ContainerStarted","Data":"8550af827ea6170856d922616c3a76a4291615298469e7a83d473cf3c22274fc"}
Apr 22 20:23:03.141796 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:03.141757    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" event={"ID":"4154cf08-4faa-45bc-91f2-7618bc433c9f","Type":"ContainerStarted","Data":"0b5a5a0e5c33369740f43942f1178b022eb786bd008690ad5bd9b993313d5f73"}
Apr 22 20:23:03.142243 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:03.141824    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:03.143334 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:03.143308    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf" event={"ID":"48359f5e-dad7-4ffa-b745-92f426d6df30","Type":"ContainerStarted","Data":"e4a0296be65882c59039416b406d8c9fab9c60ae8606d01dbb65f93fea030559"}
Apr 22 20:23:04.149410 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:04.149375    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" event={"ID":"4154cf08-4faa-45bc-91f2-7618bc433c9f","Type":"ContainerStarted","Data":"d107c5ad8ed31de7533b7d10a14df15605eb1ada133f198454648ed1aba4c307"}
Apr 22 20:23:07.165106 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:07.165068    2576 generic.go:358] "Generic (PLEG): container finished" podID="48359f5e-dad7-4ffa-b745-92f426d6df30" containerID="e4a0296be65882c59039416b406d8c9fab9c60ae8606d01dbb65f93fea030559" exitCode=0
Apr 22 20:23:07.165525 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:07.165138    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf" event={"ID":"48359f5e-dad7-4ffa-b745-92f426d6df30","Type":"ContainerDied","Data":"e4a0296be65882c59039416b406d8c9fab9c60ae8606d01dbb65f93fea030559"}
Apr 22 20:23:08.170893 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:08.170853    2576 generic.go:358] "Generic (PLEG): container finished" podID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerID="d107c5ad8ed31de7533b7d10a14df15605eb1ada133f198454648ed1aba4c307" exitCode=0
Apr 22 20:23:08.170893 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:08.170878    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" event={"ID":"4154cf08-4faa-45bc-91f2-7618bc433c9f","Type":"ContainerDied","Data":"d107c5ad8ed31de7533b7d10a14df15605eb1ada133f198454648ed1aba4c307"}
Apr 22 20:23:08.172865 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:08.172842    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf" event={"ID":"48359f5e-dad7-4ffa-b745-92f426d6df30","Type":"ContainerStarted","Data":"0133ee25d0f4ac4a2b75ef789e7ba54e66c52842ea611f1f6902a3a7a4bbf91c"}
Apr 22 20:23:08.210716 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:08.210665    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf" podStartSLOduration=7.21064571 podStartE2EDuration="7.21064571s" podCreationTimestamp="2026-04-22 20:23:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:23:08.207684975 +0000 UTC m=+1523.307641266" watchObservedRunningTime="2026-04-22 20:23:08.21064571 +0000 UTC m=+1523.310601978"
Apr 22 20:23:09.179681 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:09.179638    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" event={"ID":"4154cf08-4faa-45bc-91f2-7618bc433c9f","Type":"ContainerStarted","Data":"a343267454e9ff0063be96ceb6f43cb95ecc24f8c0bb18b722d4864b9ac03bab"}
Apr 22 20:23:09.203980 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:09.203892    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" podStartSLOduration=8.203872234 podStartE2EDuration="8.203872234s" podCreationTimestamp="2026-04-22 20:23:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:23:09.200983338 +0000 UTC m=+1524.300939603" watchObservedRunningTime="2026-04-22 20:23:09.203872234 +0000 UTC m=+1524.303828700"
Apr 22 20:23:11.917389 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:11.917352    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:11.917775 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:11.917402    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:11.918831 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:11.918802    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Apr 22 20:23:11.947245 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:11.947199    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:11.947427 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:11.947259    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:23:11.948621 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:11.948586    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf" podUID="48359f5e-dad7-4ffa-b745-92f426d6df30" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Apr 22 20:23:21.917027 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:21.916981    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Apr 22 20:23:21.934763 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:21.934727    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:23:21.947041 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:21.946993    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf" podUID="48359f5e-dad7-4ffa-b745-92f426d6df30" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Apr 22 20:23:31.917724 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:31.917664    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Apr 22 20:23:31.946747 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:31.946701    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf" podUID="48359f5e-dad7-4ffa-b745-92f426d6df30" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Apr 22 20:23:41.917891 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:41.917843    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Apr 22 20:23:41.947430 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:41.947367    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf" podUID="48359f5e-dad7-4ffa-b745-92f426d6df30" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Apr 22 20:23:51.917863 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:51.917780    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Apr 22 20:23:51.947288 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:23:51.947246    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf" podUID="48359f5e-dad7-4ffa-b745-92f426d6df30" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Apr 22 20:24:01.917017 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:24:01.916976    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Apr 22 20:24:01.946956 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:24:01.946907    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf" podUID="48359f5e-dad7-4ffa-b745-92f426d6df30" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Apr 22 20:24:11.918022 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:24:11.917979    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Apr 22 20:24:11.946921 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:24:11.946878    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf" podUID="48359f5e-dad7-4ffa-b745-92f426d6df30" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Apr 22 20:24:21.917909 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:24:21.917848    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Apr 22 20:24:21.947011 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:24:21.946968    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf" podUID="48359f5e-dad7-4ffa-b745-92f426d6df30" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Apr 22 20:24:31.917785 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:24:31.917727    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Apr 22 20:24:31.947564 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:24:31.947526    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf" podUID="48359f5e-dad7-4ffa-b745-92f426d6df30" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Apr 22 20:24:41.917404 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:24:41.917359    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Apr 22 20:24:41.947023 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:24:41.946986    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf" podUID="48359f5e-dad7-4ffa-b745-92f426d6df30" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Apr 22 20:24:51.917679 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:24:51.917618    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Apr 22 20:24:51.947404 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:24:51.947362    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf" podUID="48359f5e-dad7-4ffa-b745-92f426d6df30" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Apr 22 20:25:01.917943 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:01.917889    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Apr 22 20:25:01.947476 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:01.947436    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf" podUID="48359f5e-dad7-4ffa-b745-92f426d6df30" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Apr 22 20:25:11.917121 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:11.917073    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Apr 22 20:25:11.947383 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:11.947337    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf" podUID="48359f5e-dad7-4ffa-b745-92f426d6df30" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Apr 22 20:25:21.917935 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:21.917884    2576 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Apr 22 20:25:21.956665 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:21.956622    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:25:21.965736 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:21.965705    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:25:31.927009 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:31.926976    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:25:31.940519 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:31.940492    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:25:43.580524 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:43.580490    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"]
Apr 22 20:25:43.580926 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:43.580763    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf" podUID="48359f5e-dad7-4ffa-b745-92f426d6df30" containerName="main" containerID="cri-o://0133ee25d0f4ac4a2b75ef789e7ba54e66c52842ea611f1f6902a3a7a4bbf91c" gracePeriod=30
Apr 22 20:25:43.588164 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:43.588123    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"]
Apr 22 20:25:43.588546 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:43.588517    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="main" containerID="cri-o://a343267454e9ff0063be96ceb6f43cb95ecc24f8c0bb18b722d4864b9ac03bab" gracePeriod=30
Apr 22 20:25:58.210273 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:58.210241    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-q4nx6_9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4/istio-proxy/0.log"
Apr 22 20:25:58.236339 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:58.236311    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-6866b85949-lmpz4_bc7119c6-821e-4a5e-8827-a4c148facad7/istio-proxy/0.log"
Apr 22 20:25:58.258688 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:58.258659    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/main/0.log"
Apr 22 20:25:58.267910 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:58.267882    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/llm-d-routing-sidecar/0.log"
Apr 22 20:25:58.277506 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:58.277482    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/storage-initializer/0.log"
Apr 22 20:25:58.295755 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:58.295730    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/main/0.log"
Apr 22 20:25:58.303454 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:58.303431    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/storage-initializer/0.log"
Apr 22 20:25:59.218771 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:59.218744    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-q4nx6_9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4/istio-proxy/0.log"
Apr 22 20:25:59.235493 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:59.235466    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-6866b85949-lmpz4_bc7119c6-821e-4a5e-8827-a4c148facad7/istio-proxy/0.log"
Apr 22 20:25:59.262028 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:59.261993    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/main/0.log"
Apr 22 20:25:59.270488 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:59.270469    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/llm-d-routing-sidecar/0.log"
Apr 22 20:25:59.280184 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:59.280141    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/storage-initializer/0.log"
Apr 22 20:25:59.297607 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:59.297576    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/main/0.log"
Apr 22 20:25:59.305316 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:25:59.305294    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/storage-initializer/0.log"
Apr 22 20:26:00.223490 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:00.223460    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-q4nx6_9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4/istio-proxy/0.log"
Apr 22 20:26:00.237751 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:00.237717    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-6866b85949-lmpz4_bc7119c6-821e-4a5e-8827-a4c148facad7/istio-proxy/0.log"
Apr 22 20:26:00.260391 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:00.260361    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/main/0.log"
Apr 22 20:26:00.267418 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:00.267399    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/llm-d-routing-sidecar/0.log"
Apr 22 20:26:00.277243 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:00.277219    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/storage-initializer/0.log"
Apr 22 20:26:00.295738 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:00.295700    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/main/0.log"
Apr 22 20:26:00.302762 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:00.302735    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/storage-initializer/0.log"
Apr 22 20:26:01.196806 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:01.196775    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-q4nx6_9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4/istio-proxy/0.log"
Apr 22 20:26:01.212560 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:01.212531    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-6866b85949-lmpz4_bc7119c6-821e-4a5e-8827-a4c148facad7/istio-proxy/0.log"
Apr 22 20:26:01.231670 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:01.231639    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/main/0.log"
Apr 22 20:26:01.237974 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:01.237940    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/llm-d-routing-sidecar/0.log"
Apr 22 20:26:01.247654 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:01.247634    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/storage-initializer/0.log"
Apr 22 20:26:01.262950 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:01.262921    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/main/0.log"
Apr 22 20:26:01.271011 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:01.270989    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/storage-initializer/0.log"
Apr 22 20:26:02.162940 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:02.162862    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-q4nx6_9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4/istio-proxy/0.log"
Apr 22 20:26:02.178425 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:02.178393    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-6866b85949-lmpz4_bc7119c6-821e-4a5e-8827-a4c148facad7/istio-proxy/0.log"
Apr 22 20:26:02.199312 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:02.199292    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/main/0.log"
Apr 22 20:26:02.210605 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:02.210586    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/llm-d-routing-sidecar/0.log"
Apr 22 20:26:02.220387 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:02.220370    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/storage-initializer/0.log"
Apr 22 20:26:02.237497 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:02.237472    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/main/0.log"
Apr 22 20:26:02.245241 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:02.245218    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/storage-initializer/0.log"
Apr 22 20:26:03.127377 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:03.127352    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-q4nx6_9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4/istio-proxy/0.log"
Apr 22 20:26:03.142596 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:03.142567    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-6866b85949-lmpz4_bc7119c6-821e-4a5e-8827-a4c148facad7/istio-proxy/0.log"
Apr 22 20:26:03.161800 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:03.161771    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/main/0.log"
Apr 22 20:26:03.167855 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:03.167833    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/llm-d-routing-sidecar/0.log"
Apr 22 20:26:03.177683 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:03.177667    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/storage-initializer/0.log"
Apr 22 20:26:03.195689 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:03.195654    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/main/0.log"
Apr 22 20:26:03.203514 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:03.203497    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/storage-initializer/0.log"
Apr 22 20:26:04.088275 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:04.088249    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-q4nx6_9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4/istio-proxy/0.log"
Apr 22 20:26:04.105546 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:04.105524    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-6866b85949-lmpz4_bc7119c6-821e-4a5e-8827-a4c148facad7/istio-proxy/0.log"
Apr 22 20:26:04.138296 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:04.138272    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/main/0.log"
Apr 22 20:26:04.144722 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:04.144705    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/llm-d-routing-sidecar/0.log"
Apr 22 20:26:04.154222 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:04.154205    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/storage-initializer/0.log"
Apr 22 20:26:04.170326 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:04.170310    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/main/0.log"
Apr 22 20:26:04.177645 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:04.177629    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/storage-initializer/0.log"
Apr 22 20:26:05.079266 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:05.079233    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-q4nx6_9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4/istio-proxy/0.log"
Apr 22 20:26:05.093512 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:05.093489    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-6866b85949-lmpz4_bc7119c6-821e-4a5e-8827-a4c148facad7/istio-proxy/0.log"
Apr 22 20:26:05.113962 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:05.113938    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/main/0.log"
Apr 22 20:26:05.120321 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:05.120299    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/llm-d-routing-sidecar/0.log"
Apr 22 20:26:05.130383 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:05.130360    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/storage-initializer/0.log"
Apr 22 20:26:05.148527 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:05.148506    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/main/0.log"
Apr 22 20:26:05.157028 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:05.157007    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/storage-initializer/0.log"
Apr 22 20:26:06.092473 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:06.092446    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-q4nx6_9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4/istio-proxy/0.log"
Apr 22 20:26:06.107409 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:06.107388    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-6866b85949-lmpz4_bc7119c6-821e-4a5e-8827-a4c148facad7/istio-proxy/0.log"
Apr 22 20:26:06.130130 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:06.130108    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/main/0.log"
Apr 22 20:26:06.137685 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:06.137669    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/llm-d-routing-sidecar/0.log"
Apr 22 20:26:06.148709 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:06.148686    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/storage-initializer/0.log"
Apr 22 20:26:06.166086 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:06.166062    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/main/0.log"
Apr 22 20:26:06.176723 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:06.176700    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/storage-initializer/0.log"
Apr 22 20:26:07.146211 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:07.146143    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-q4nx6_9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4/istio-proxy/0.log"
Apr 22 20:26:07.163724 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:07.163698    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-6866b85949-lmpz4_bc7119c6-821e-4a5e-8827-a4c148facad7/istio-proxy/0.log"
Apr 22 20:26:07.186635 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:07.186603    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/main/0.log"
Apr 22 20:26:07.193132 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:07.193109    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/llm-d-routing-sidecar/0.log"
Apr 22 20:26:07.202965 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:07.202941    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/storage-initializer/0.log"
Apr 22 20:26:07.218960 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:07.218931    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/main/0.log"
Apr 22 20:26:07.226575 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:07.226553    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/storage-initializer/0.log"
Apr 22 20:26:08.129895 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:08.129864    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-q4nx6_9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4/istio-proxy/0.log"
Apr 22 20:26:08.145562 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:08.145533    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-6866b85949-lmpz4_bc7119c6-821e-4a5e-8827-a4c148facad7/istio-proxy/0.log"
Apr 22 20:26:08.166569 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:08.166544    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/main/0.log"
Apr 22 20:26:08.172952 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:08.172933    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/llm-d-routing-sidecar/0.log"
Apr 22 20:26:08.183493 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:08.183471    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/storage-initializer/0.log"
Apr 22 20:26:08.200663 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:08.200629    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/main/0.log"
Apr 22 20:26:08.207853 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:08.207835    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/storage-initializer/0.log"
Apr 22 20:26:09.119402 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:09.119378    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-q4nx6_9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4/istio-proxy/0.log"
Apr 22 20:26:09.133831 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:09.133811    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-6866b85949-lmpz4_bc7119c6-821e-4a5e-8827-a4c148facad7/istio-proxy/0.log"
Apr 22 20:26:09.154163 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:09.154133    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/main/0.log"
Apr 22 20:26:09.161568 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:09.161546    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/llm-d-routing-sidecar/0.log"
Apr 22 20:26:09.173476 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:09.173456    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/storage-initializer/0.log"
Apr 22 20:26:09.190785 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:09.190767    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/main/0.log"
Apr 22 20:26:09.199435 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:09.199418    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/storage-initializer/0.log"
Apr 22 20:26:10.124516 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:10.124492    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-q4nx6_9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4/istio-proxy/0.log"
Apr 22 20:26:10.141196 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:10.141141    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-6866b85949-lmpz4_bc7119c6-821e-4a5e-8827-a4c148facad7/istio-proxy/0.log"
Apr 22 20:26:10.162516 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:10.162488    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/main/0.log"
Apr 22 20:26:10.168792 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:10.168769    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/llm-d-routing-sidecar/0.log"
Apr 22 20:26:10.178277 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:10.178258    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/storage-initializer/0.log"
Apr 22 20:26:10.196351 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:10.196331    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/main/0.log"
Apr 22 20:26:10.205879 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:10.205860    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/storage-initializer/0.log"
Apr 22 20:26:11.104078 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:11.104049    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-6c59fbf55c-q4nx6_9482a8d9-c8e8-4a15-a8b8-c5f6e351f9e4/istio-proxy/0.log"
Apr 22 20:26:11.118950 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:11.118915    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-6866b85949-lmpz4_bc7119c6-821e-4a5e-8827-a4c148facad7/istio-proxy/0.log"
Apr 22 20:26:11.138348 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:11.138324    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/main/0.log"
Apr 22 20:26:11.144846 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:11.144823    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/llm-d-routing-sidecar/0.log"
Apr 22 20:26:11.155960 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:11.155937    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/storage-initializer/0.log"
Apr 22 20:26:11.171830 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:11.171803    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/main/0.log"
Apr 22 20:26:11.180238 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:11.180214    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf_48359f5e-dad7-4ffa-b745-92f426d6df30/storage-initializer/0.log"
Apr 22 20:26:12.110221 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:12.110187    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-55ff986f96-92qlr_a36f74c3-7c15-47c1-baaa-4043e58c120c/discovery/0.log"
Apr 22 20:26:12.122089 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:12.122056    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-7c5447bb76-w9nb5_1cf701c3-b578-4cc8-a6af-4084c426e3ef/istio-proxy/0.log"
Apr 22 20:26:12.904166 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:12.904114    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-55ff986f96-92qlr_a36f74c3-7c15-47c1-baaa-4043e58c120c/discovery/0.log"
Apr 22 20:26:12.916547 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:12.916525    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-7c5447bb76-w9nb5_1cf701c3-b578-4cc8-a6af-4084c426e3ef/istio-proxy/0.log"
Apr 22 20:26:13.589052 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.588987    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="llm-d-routing-sidecar" containerID="cri-o://0b5a5a0e5c33369740f43942f1178b022eb786bd008690ad5bd9b993313d5f73" gracePeriod=2
Apr 22 20:26:13.747105 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.747036    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-khzl8_302ecd8f-e911-4d5b-b494-b1893a7fc59e/manager/0.log"
Apr 22 20:26:13.799689 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.799660    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6ddf9554fc-8tvn5_6f2155bb-f25e-4ab1-b66a-77d84bb92aac/manager/0.log"
Apr 22 20:26:13.826594 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.826563    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-skkk9_d733567f-ee6d-4c59-9402-d34f72845cad/manager/0.log"
Apr 22 20:26:13.883242 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.883221    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:26:13.886381 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.886363    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/main/0.log"
Apr 22 20:26:13.886950 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.886935    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:26:13.994000 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.993959    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-kserve-provision-location\") pod \"4154cf08-4faa-45bc-91f2-7618bc433c9f\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") "
Apr 22 20:26:13.994515 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.994047    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-model-cache\") pod \"48359f5e-dad7-4ffa-b745-92f426d6df30\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") "
Apr 22 20:26:13.994515 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.994072    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-dshm\") pod \"4154cf08-4faa-45bc-91f2-7618bc433c9f\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") "
Apr 22 20:26:13.994515 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.994104    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-home\") pod \"48359f5e-dad7-4ffa-b745-92f426d6df30\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") "
Apr 22 20:26:13.994515 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.994183    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-model-cache\") pod \"4154cf08-4faa-45bc-91f2-7618bc433c9f\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") "
Apr 22 20:26:13.994515 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.994227    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-dshm\") pod \"48359f5e-dad7-4ffa-b745-92f426d6df30\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") "
Apr 22 20:26:13.994515 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.994256    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gndz\" (UniqueName: \"kubernetes.io/projected/4154cf08-4faa-45bc-91f2-7618bc433c9f-kube-api-access-9gndz\") pod \"4154cf08-4faa-45bc-91f2-7618bc433c9f\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") "
Apr 22 20:26:13.994515 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.994344    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-kserve-provision-location\") pod \"48359f5e-dad7-4ffa-b745-92f426d6df30\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") "
Apr 22 20:26:13.994515 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.994379    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4gfr\" (UniqueName: \"kubernetes.io/projected/48359f5e-dad7-4ffa-b745-92f426d6df30-kube-api-access-d4gfr\") pod \"48359f5e-dad7-4ffa-b745-92f426d6df30\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") "
Apr 22 20:26:13.994515 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.994409    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4154cf08-4faa-45bc-91f2-7618bc433c9f-tls-certs\") pod \"4154cf08-4faa-45bc-91f2-7618bc433c9f\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") "
Apr 22 20:26:13.994515 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.994433    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-home\") pod \"4154cf08-4faa-45bc-91f2-7618bc433c9f\" (UID: \"4154cf08-4faa-45bc-91f2-7618bc433c9f\") "
Apr 22 20:26:13.994515 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.994454    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48359f5e-dad7-4ffa-b745-92f426d6df30-tls-certs\") pod \"48359f5e-dad7-4ffa-b745-92f426d6df30\" (UID: \"48359f5e-dad7-4ffa-b745-92f426d6df30\") "
Apr 22 20:26:13.995062 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.994400    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-model-cache" (OuterVolumeSpecName: "model-cache") pod "4154cf08-4faa-45bc-91f2-7618bc433c9f" (UID: "4154cf08-4faa-45bc-91f2-7618bc433c9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:26:13.995062 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.994685    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-model-cache" (OuterVolumeSpecName: "model-cache") pod "48359f5e-dad7-4ffa-b745-92f426d6df30" (UID: "48359f5e-dad7-4ffa-b745-92f426d6df30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:26:13.995188 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.995138    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-home" (OuterVolumeSpecName: "home") pod "4154cf08-4faa-45bc-91f2-7618bc433c9f" (UID: "4154cf08-4faa-45bc-91f2-7618bc433c9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:26:13.995302 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.995281    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-home\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:26:13.995366 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.995309    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-model-cache\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:26:13.995366 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.995325    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-model-cache\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:26:13.997164 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.997124    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-dshm" (OuterVolumeSpecName: "dshm") pod "48359f5e-dad7-4ffa-b745-92f426d6df30" (UID: "48359f5e-dad7-4ffa-b745-92f426d6df30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:26:13.997325 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.997304    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48359f5e-dad7-4ffa-b745-92f426d6df30-kube-api-access-d4gfr" (OuterVolumeSpecName: "kube-api-access-d4gfr") pod "48359f5e-dad7-4ffa-b745-92f426d6df30" (UID: "48359f5e-dad7-4ffa-b745-92f426d6df30"). InnerVolumeSpecName "kube-api-access-d4gfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:26:13.997397 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.997369    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-dshm" (OuterVolumeSpecName: "dshm") pod "4154cf08-4faa-45bc-91f2-7618bc433c9f" (UID: "4154cf08-4faa-45bc-91f2-7618bc433c9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:26:13.997440 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.997391    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4154cf08-4faa-45bc-91f2-7618bc433c9f-kube-api-access-9gndz" (OuterVolumeSpecName: "kube-api-access-9gndz") pod "4154cf08-4faa-45bc-91f2-7618bc433c9f" (UID: "4154cf08-4faa-45bc-91f2-7618bc433c9f"). InnerVolumeSpecName "kube-api-access-9gndz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:26:13.998210 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.998183    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-home" (OuterVolumeSpecName: "home") pod "48359f5e-dad7-4ffa-b745-92f426d6df30" (UID: "48359f5e-dad7-4ffa-b745-92f426d6df30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:26:13.998454 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.998429    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4154cf08-4faa-45bc-91f2-7618bc433c9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4154cf08-4faa-45bc-91f2-7618bc433c9f" (UID: "4154cf08-4faa-45bc-91f2-7618bc433c9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:26:13.998767 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:13.998750    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48359f5e-dad7-4ffa-b745-92f426d6df30-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48359f5e-dad7-4ffa-b745-92f426d6df30" (UID: "48359f5e-dad7-4ffa-b745-92f426d6df30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:26:14.022324 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.022289    2576 generic.go:358] "Generic (PLEG): container finished" podID="48359f5e-dad7-4ffa-b745-92f426d6df30" containerID="0133ee25d0f4ac4a2b75ef789e7ba54e66c52842ea611f1f6902a3a7a4bbf91c" exitCode=137
Apr 22 20:26:14.022507 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.022332    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf" event={"ID":"48359f5e-dad7-4ffa-b745-92f426d6df30","Type":"ContainerDied","Data":"0133ee25d0f4ac4a2b75ef789e7ba54e66c52842ea611f1f6902a3a7a4bbf91c"}
Apr 22 20:26:14.022507 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.022379    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"
Apr 22 20:26:14.022507 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.022395    2576 scope.go:117] "RemoveContainer" containerID="0133ee25d0f4ac4a2b75ef789e7ba54e66c52842ea611f1f6902a3a7a4bbf91c"
Apr 22 20:26:14.022507 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.022382    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf" event={"ID":"48359f5e-dad7-4ffa-b745-92f426d6df30","Type":"ContainerDied","Data":"fa356dc228f24936054aed23ee1feccea795b24a11872a0fedfc6ddd654756d4"}
Apr 22 20:26:14.023930 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.023913    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54b886bf4d-vdz6d_4154cf08-4faa-45bc-91f2-7618bc433c9f/main/0.log"
Apr 22 20:26:14.024552 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.024530    2576 generic.go:358] "Generic (PLEG): container finished" podID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerID="a343267454e9ff0063be96ceb6f43cb95ecc24f8c0bb18b722d4864b9ac03bab" exitCode=137
Apr 22 20:26:14.024552 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.024553    2576 generic.go:358] "Generic (PLEG): container finished" podID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerID="0b5a5a0e5c33369740f43942f1178b022eb786bd008690ad5bd9b993313d5f73" exitCode=0
Apr 22 20:26:14.024692 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.024617    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"
Apr 22 20:26:14.024692 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.024630    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" event={"ID":"4154cf08-4faa-45bc-91f2-7618bc433c9f","Type":"ContainerDied","Data":"a343267454e9ff0063be96ceb6f43cb95ecc24f8c0bb18b722d4864b9ac03bab"}
Apr 22 20:26:14.024692 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.024662    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" event={"ID":"4154cf08-4faa-45bc-91f2-7618bc433c9f","Type":"ContainerDied","Data":"0b5a5a0e5c33369740f43942f1178b022eb786bd008690ad5bd9b993313d5f73"}
Apr 22 20:26:14.024692 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.024679    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d" event={"ID":"4154cf08-4faa-45bc-91f2-7618bc433c9f","Type":"ContainerDied","Data":"8550af827ea6170856d922616c3a76a4291615298469e7a83d473cf3c22274fc"}
Apr 22 20:26:14.044254 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.044216    2576 scope.go:117] "RemoveContainer" containerID="e4a0296be65882c59039416b406d8c9fab9c60ae8606d01dbb65f93fea030559"
Apr 22 20:26:14.067108 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.067055    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "48359f5e-dad7-4ffa-b745-92f426d6df30" (UID: "48359f5e-dad7-4ffa-b745-92f426d6df30"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:26:14.070520 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.070483    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4154cf08-4faa-45bc-91f2-7618bc433c9f" (UID: "4154cf08-4faa-45bc-91f2-7618bc433c9f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:26:14.096514 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.096477    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-dshm\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:26:14.096514 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.096516    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gndz\" (UniqueName: \"kubernetes.io/projected/4154cf08-4faa-45bc-91f2-7618bc433c9f-kube-api-access-9gndz\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:26:14.096643 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.096532    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-kserve-provision-location\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:26:14.096643 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.096546    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4gfr\" (UniqueName: \"kubernetes.io/projected/48359f5e-dad7-4ffa-b745-92f426d6df30-kube-api-access-d4gfr\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:26:14.096643 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.096560    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4154cf08-4faa-45bc-91f2-7618bc433c9f-tls-certs\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:26:14.096643 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.096572    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48359f5e-dad7-4ffa-b745-92f426d6df30-tls-certs\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:26:14.096643 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.096585    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-kserve-provision-location\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:26:14.096643 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.096599    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4154cf08-4faa-45bc-91f2-7618bc433c9f-dshm\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:26:14.096643 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.096610    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48359f5e-dad7-4ffa-b745-92f426d6df30-home\") on node \"ip-10-0-135-245.ec2.internal\" DevicePath \"\""
Apr 22 20:26:14.121862 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.121833    2576 scope.go:117] "RemoveContainer" containerID="0133ee25d0f4ac4a2b75ef789e7ba54e66c52842ea611f1f6902a3a7a4bbf91c"
Apr 22 20:26:14.122252 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:26:14.122226    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0133ee25d0f4ac4a2b75ef789e7ba54e66c52842ea611f1f6902a3a7a4bbf91c\": container with ID starting with 0133ee25d0f4ac4a2b75ef789e7ba54e66c52842ea611f1f6902a3a7a4bbf91c not found: ID does not exist" containerID="0133ee25d0f4ac4a2b75ef789e7ba54e66c52842ea611f1f6902a3a7a4bbf91c"
Apr 22 20:26:14.122336 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.122264    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0133ee25d0f4ac4a2b75ef789e7ba54e66c52842ea611f1f6902a3a7a4bbf91c"} err="failed to get container status \"0133ee25d0f4ac4a2b75ef789e7ba54e66c52842ea611f1f6902a3a7a4bbf91c\": rpc error: code = NotFound desc = could not find container \"0133ee25d0f4ac4a2b75ef789e7ba54e66c52842ea611f1f6902a3a7a4bbf91c\": container with ID starting with 0133ee25d0f4ac4a2b75ef789e7ba54e66c52842ea611f1f6902a3a7a4bbf91c not found: ID does not exist"
Apr 22 20:26:14.122336 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.122285    2576 scope.go:117] "RemoveContainer" containerID="e4a0296be65882c59039416b406d8c9fab9c60ae8606d01dbb65f93fea030559"
Apr 22 20:26:14.122561 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:26:14.122533    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4a0296be65882c59039416b406d8c9fab9c60ae8606d01dbb65f93fea030559\": container with ID starting with e4a0296be65882c59039416b406d8c9fab9c60ae8606d01dbb65f93fea030559 not found: ID does not exist" containerID="e4a0296be65882c59039416b406d8c9fab9c60ae8606d01dbb65f93fea030559"
Apr 22 20:26:14.122632 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.122571    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a0296be65882c59039416b406d8c9fab9c60ae8606d01dbb65f93fea030559"} err="failed to get container status \"e4a0296be65882c59039416b406d8c9fab9c60ae8606d01dbb65f93fea030559\": rpc error: code = NotFound desc = could not find container \"e4a0296be65882c59039416b406d8c9fab9c60ae8606d01dbb65f93fea030559\": container with ID starting with e4a0296be65882c59039416b406d8c9fab9c60ae8606d01dbb65f93fea030559 not found: ID does not exist"
Apr 22 20:26:14.122632 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.122600    2576 scope.go:117] "RemoveContainer" containerID="a343267454e9ff0063be96ceb6f43cb95ecc24f8c0bb18b722d4864b9ac03bab"
Apr 22 20:26:14.143255 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.143235    2576 scope.go:117] "RemoveContainer" containerID="d107c5ad8ed31de7533b7d10a14df15605eb1ada133f198454648ed1aba4c307"
Apr 22 20:26:14.212337 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.212308    2576 scope.go:117] "RemoveContainer" containerID="0b5a5a0e5c33369740f43942f1178b022eb786bd008690ad5bd9b993313d5f73"
Apr 22 20:26:14.221061 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.221043    2576 scope.go:117] "RemoveContainer" containerID="a343267454e9ff0063be96ceb6f43cb95ecc24f8c0bb18b722d4864b9ac03bab"
Apr 22 20:26:14.221360 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:26:14.221330    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a343267454e9ff0063be96ceb6f43cb95ecc24f8c0bb18b722d4864b9ac03bab\": container with ID starting with a343267454e9ff0063be96ceb6f43cb95ecc24f8c0bb18b722d4864b9ac03bab not found: ID does not exist" containerID="a343267454e9ff0063be96ceb6f43cb95ecc24f8c0bb18b722d4864b9ac03bab"
Apr 22 20:26:14.221446 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.221372    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a343267454e9ff0063be96ceb6f43cb95ecc24f8c0bb18b722d4864b9ac03bab"} err="failed to get container status \"a343267454e9ff0063be96ceb6f43cb95ecc24f8c0bb18b722d4864b9ac03bab\": rpc error: code = NotFound desc = could not find container \"a343267454e9ff0063be96ceb6f43cb95ecc24f8c0bb18b722d4864b9ac03bab\": container with ID starting with a343267454e9ff0063be96ceb6f43cb95ecc24f8c0bb18b722d4864b9ac03bab not found: ID does not exist"
Apr 22 20:26:14.221446 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.221399    2576 scope.go:117] "RemoveContainer" containerID="d107c5ad8ed31de7533b7d10a14df15605eb1ada133f198454648ed1aba4c307"
Apr 22 20:26:14.221642 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:26:14.221626    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d107c5ad8ed31de7533b7d10a14df15605eb1ada133f198454648ed1aba4c307\": container with ID starting with d107c5ad8ed31de7533b7d10a14df15605eb1ada133f198454648ed1aba4c307 not found: ID does not exist" containerID="d107c5ad8ed31de7533b7d10a14df15605eb1ada133f198454648ed1aba4c307"
Apr 22 20:26:14.221681 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.221649    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d107c5ad8ed31de7533b7d10a14df15605eb1ada133f198454648ed1aba4c307"} err="failed to get container status \"d107c5ad8ed31de7533b7d10a14df15605eb1ada133f198454648ed1aba4c307\": rpc error: code = NotFound desc = could not find container \"d107c5ad8ed31de7533b7d10a14df15605eb1ada133f198454648ed1aba4c307\": container with ID starting with d107c5ad8ed31de7533b7d10a14df15605eb1ada133f198454648ed1aba4c307 not found: ID does not exist"
Apr 22 20:26:14.221681 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.221665    2576 scope.go:117] "RemoveContainer" containerID="0b5a5a0e5c33369740f43942f1178b022eb786bd008690ad5bd9b993313d5f73"
Apr 22 20:26:14.221869 ip-10-0-135-245 kubenswrapper[2576]: E0422 20:26:14.221845    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b5a5a0e5c33369740f43942f1178b022eb786bd008690ad5bd9b993313d5f73\": container with ID starting with 0b5a5a0e5c33369740f43942f1178b022eb786bd008690ad5bd9b993313d5f73 not found: ID does not exist" containerID="0b5a5a0e5c33369740f43942f1178b022eb786bd008690ad5bd9b993313d5f73"
Apr 22 20:26:14.221972 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.221873    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b5a5a0e5c33369740f43942f1178b022eb786bd008690ad5bd9b993313d5f73"} err="failed to get container status \"0b5a5a0e5c33369740f43942f1178b022eb786bd008690ad5bd9b993313d5f73\": rpc error: code = NotFound desc = could not find container \"0b5a5a0e5c33369740f43942f1178b022eb786bd008690ad5bd9b993313d5f73\": container with ID starting with 0b5a5a0e5c33369740f43942f1178b022eb786bd008690ad5bd9b993313d5f73 not found: ID does not exist"
Apr 22 20:26:14.221972 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.221888    2576 scope.go:117] "RemoveContainer" containerID="a343267454e9ff0063be96ceb6f43cb95ecc24f8c0bb18b722d4864b9ac03bab"
Apr 22 20:26:14.222115 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.222095    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a343267454e9ff0063be96ceb6f43cb95ecc24f8c0bb18b722d4864b9ac03bab"} err="failed to get container status \"a343267454e9ff0063be96ceb6f43cb95ecc24f8c0bb18b722d4864b9ac03bab\": rpc error: code = NotFound desc = could not find container \"a343267454e9ff0063be96ceb6f43cb95ecc24f8c0bb18b722d4864b9ac03bab\": container with ID starting with a343267454e9ff0063be96ceb6f43cb95ecc24f8c0bb18b722d4864b9ac03bab not found: ID does not exist"
Apr 22 20:26:14.222217 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.222118    2576 scope.go:117] "RemoveContainer" containerID="d107c5ad8ed31de7533b7d10a14df15605eb1ada133f198454648ed1aba4c307"
Apr 22 20:26:14.222362 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.222343    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d107c5ad8ed31de7533b7d10a14df15605eb1ada133f198454648ed1aba4c307"} err="failed to get container status \"d107c5ad8ed31de7533b7d10a14df15605eb1ada133f198454648ed1aba4c307\": rpc error: code = NotFound desc = could not find container \"d107c5ad8ed31de7533b7d10a14df15605eb1ada133f198454648ed1aba4c307\": container with ID starting with d107c5ad8ed31de7533b7d10a14df15605eb1ada133f198454648ed1aba4c307 not found: ID does not exist"
Apr 22 20:26:14.222402 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.222364    2576 scope.go:117] "RemoveContainer" containerID="0b5a5a0e5c33369740f43942f1178b022eb786bd008690ad5bd9b993313d5f73"
Apr 22 20:26:14.222560 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.222540    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b5a5a0e5c33369740f43942f1178b022eb786bd008690ad5bd9b993313d5f73"} err="failed to get container status \"0b5a5a0e5c33369740f43942f1178b022eb786bd008690ad5bd9b993313d5f73\": rpc error: code = NotFound desc = could not find container \"0b5a5a0e5c33369740f43942f1178b022eb786bd008690ad5bd9b993313d5f73\": container with ID starting with 0b5a5a0e5c33369740f43942f1178b022eb786bd008690ad5bd9b993313d5f73 not found: ID does not exist"
Apr 22 20:26:14.347198 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.347141    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"]
Apr 22 20:26:14.349879 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.349853    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-57c6dbf7f7-6q7sf"]
Apr 22 20:26:14.359538 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.359515    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"]
Apr 22 20:26:14.371267 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:14.371240    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54b886bf4d-vdz6d"]
Apr 22 20:26:15.374704 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:15.374669    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" path="/var/lib/kubelet/pods/4154cf08-4faa-45bc-91f2-7618bc433c9f/volumes"
Apr 22 20:26:15.375124 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:15.375110    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48359f5e-dad7-4ffa-b745-92f426d6df30" path="/var/lib/kubelet/pods/48359f5e-dad7-4ffa-b745-92f426d6df30/volumes"
Apr 22 20:26:18.787379 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:18.787351    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d64wb_3d9b1fb8-25a2-4fc7-8083-c6e1ee790339/global-pull-secret-syncer/0.log"
Apr 22 20:26:18.846008 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:18.845983    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pmrkj_d5d3cbed-7c79-4ea8-a02c-2992c80af473/konnectivity-agent/0.log"
Apr 22 20:26:18.939859 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:18.939831    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-245.ec2.internal_0353ff154730f2cf2a2ea68e586089e5/haproxy/0.log"
Apr 22 20:26:23.216060 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:23.216026    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-khzl8_302ecd8f-e911-4d5b-b494-b1893a7fc59e/manager/0.log"
Apr 22 20:26:23.291907 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:23.291877    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6ddf9554fc-8tvn5_6f2155bb-f25e-4ab1-b66a-77d84bb92aac/manager/0.log"
Apr 22 20:26:23.339529 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:23.339502    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-skkk9_d733567f-ee6d-4c59-9402-d34f72845cad/manager/0.log"
Apr 22 20:26:24.303543 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:24.303515    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_92c880f0-b8a6-4dfb-a70c-bae6ec61f048/alertmanager/0.log"
Apr 22 20:26:24.331906 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:24.331881    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_92c880f0-b8a6-4dfb-a70c-bae6ec61f048/config-reloader/0.log"
Apr 22 20:26:24.352728 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:24.352709    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_92c880f0-b8a6-4dfb-a70c-bae6ec61f048/kube-rbac-proxy-web/0.log"
Apr 22 20:26:24.376912 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:24.376894    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_92c880f0-b8a6-4dfb-a70c-bae6ec61f048/kube-rbac-proxy/0.log"
Apr 22 20:26:24.397773 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:24.397751    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_92c880f0-b8a6-4dfb-a70c-bae6ec61f048/kube-rbac-proxy-metric/0.log"
Apr 22 20:26:24.420949 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:24.420928    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_92c880f0-b8a6-4dfb-a70c-bae6ec61f048/prom-label-proxy/0.log"
Apr 22 20:26:24.447359 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:24.447340    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_92c880f0-b8a6-4dfb-a70c-bae6ec61f048/init-config-reloader/0.log"
Apr 22 20:26:24.649191 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:24.649167    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-24q4t_d3024bfe-c134-4366-9106-da71be312c10/node-exporter/0.log"
Apr 22 20:26:24.671773 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:24.671751    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-24q4t_d3024bfe-c134-4366-9106-da71be312c10/kube-rbac-proxy/0.log"
Apr 22 20:26:24.694464 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:24.694442    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-24q4t_d3024bfe-c134-4366-9106-da71be312c10/init-textfile/0.log"
Apr 22 20:26:24.869828 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:24.869799    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9d44df66c-cz8f5_ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb/kube-rbac-proxy-main/0.log"
Apr 22 20:26:24.890390 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:24.890369    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9d44df66c-cz8f5_ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb/kube-rbac-proxy-self/0.log"
Apr 22 20:26:24.912460 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:24.912394    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9d44df66c-cz8f5_ae7ecfaa-5ca6-4e9a-9a2c-23f8946a38cb/openshift-state-metrics/0.log"
Apr 22 20:26:25.170663 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:25.170582    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-57cf98b594-pp5hw_d8f4ac74-fc98-4d51-8bb7-136b4695a7bb/prometheus-operator-admission-webhook/0.log"
Apr 22 20:26:25.306117 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:25.306076    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5665d7f4f-w8kfx_825d8bf0-4414-42c0-bf95-61d4ee25dc9a/thanos-query/0.log"
Apr 22 20:26:25.326535 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:25.326509    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5665d7f4f-w8kfx_825d8bf0-4414-42c0-bf95-61d4ee25dc9a/kube-rbac-proxy-web/0.log"
Apr 22 20:26:25.352285 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:25.352261    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5665d7f4f-w8kfx_825d8bf0-4414-42c0-bf95-61d4ee25dc9a/kube-rbac-proxy/0.log"
Apr 22 20:26:25.374907 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:25.374881    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5665d7f4f-w8kfx_825d8bf0-4414-42c0-bf95-61d4ee25dc9a/prom-label-proxy/0.log"
Apr 22 20:26:25.395820 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:25.395803    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5665d7f4f-w8kfx_825d8bf0-4414-42c0-bf95-61d4ee25dc9a/kube-rbac-proxy-rules/0.log"
Apr 22 20:26:25.416108 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:25.416087    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5665d7f4f-w8kfx_825d8bf0-4414-42c0-bf95-61d4ee25dc9a/kube-rbac-proxy-metrics/0.log"
Apr 22 20:26:27.558084 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.558050    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5bd97b97cc-rnh5p_53185718-efed-4ae5-a2d0-6e0e4301109c/console/0.log"
Apr 22 20:26:27.588797 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.588771    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6bcc868b7-fjzng_cf9153e8-d813-4e2c-bd0a-1140521bf7df/download-server/0.log"
Apr 22 20:26:27.772217 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.772187    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"]
Apr 22 20:26:27.772609 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.772596    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48359f5e-dad7-4ffa-b745-92f426d6df30" containerName="main"
Apr 22 20:26:27.772660 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.772612    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="48359f5e-dad7-4ffa-b745-92f426d6df30" containerName="main"
Apr 22 20:26:27.772660 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.772632    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="storage-initializer"
Apr 22 20:26:27.772660 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.772638    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="storage-initializer"
Apr 22 20:26:27.772660 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.772649    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="main"
Apr 22 20:26:27.772660 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.772654    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="main"
Apr 22 20:26:27.772813 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.772663    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="llm-d-routing-sidecar"
Apr 22 20:26:27.772813 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.772668    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="llm-d-routing-sidecar"
Apr 22 20:26:27.772813 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.772673    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48359f5e-dad7-4ffa-b745-92f426d6df30" containerName="storage-initializer"
Apr 22 20:26:27.772813 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.772679    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="48359f5e-dad7-4ffa-b745-92f426d6df30" containerName="storage-initializer"
Apr 22 20:26:27.772813 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.772750    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="main"
Apr 22 20:26:27.772813 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.772760    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="4154cf08-4faa-45bc-91f2-7618bc433c9f" containerName="llm-d-routing-sidecar"
Apr 22 20:26:27.772813 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.772768    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="48359f5e-dad7-4ffa-b745-92f426d6df30" containerName="main"
Apr 22 20:26:27.786979 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.786945    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"]
Apr 22 20:26:27.787130 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.787056    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"
Apr 22 20:26:27.789279 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.789255    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7cxtv\"/\"openshift-service-ca.crt\""
Apr 22 20:26:27.789878 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.789858    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7cxtv\"/\"default-dockercfg-phjf8\""
Apr 22 20:26:27.789980 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.789863    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7cxtv\"/\"kube-root-ca.crt\""
Apr 22 20:26:27.926584 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.926537    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/129ecb30-98f2-42e5-ba80-98e6c4994ae9-sys\") pod \"perf-node-gather-daemonset-jn4z6\" (UID: \"129ecb30-98f2-42e5-ba80-98e6c4994ae9\") " pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"
Apr 22 20:26:27.926584 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.926592    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/129ecb30-98f2-42e5-ba80-98e6c4994ae9-lib-modules\") pod \"perf-node-gather-daemonset-jn4z6\" (UID: \"129ecb30-98f2-42e5-ba80-98e6c4994ae9\") " pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"
Apr 22 20:26:27.926816 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.926686    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/129ecb30-98f2-42e5-ba80-98e6c4994ae9-proc\") pod \"perf-node-gather-daemonset-jn4z6\" (UID: \"129ecb30-98f2-42e5-ba80-98e6c4994ae9\") " pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"
Apr 22 20:26:27.926816 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.926719    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s9xq\" (UniqueName: \"kubernetes.io/projected/129ecb30-98f2-42e5-ba80-98e6c4994ae9-kube-api-access-9s9xq\") pod \"perf-node-gather-daemonset-jn4z6\" (UID: \"129ecb30-98f2-42e5-ba80-98e6c4994ae9\") " pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"
Apr 22 20:26:27.926816 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:27.926743    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/129ecb30-98f2-42e5-ba80-98e6c4994ae9-podres\") pod \"perf-node-gather-daemonset-jn4z6\" (UID: \"129ecb30-98f2-42e5-ba80-98e6c4994ae9\") " pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"
Apr 22 20:26:28.028112 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:28.028061    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/129ecb30-98f2-42e5-ba80-98e6c4994ae9-sys\") pod \"perf-node-gather-daemonset-jn4z6\" (UID: \"129ecb30-98f2-42e5-ba80-98e6c4994ae9\") " pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"
Apr 22 20:26:28.028112 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:28.028116    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/129ecb30-98f2-42e5-ba80-98e6c4994ae9-lib-modules\") pod \"perf-node-gather-daemonset-jn4z6\" (UID: \"129ecb30-98f2-42e5-ba80-98e6c4994ae9\") " pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"
Apr 22 20:26:28.028370 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:28.028204    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/129ecb30-98f2-42e5-ba80-98e6c4994ae9-proc\") pod \"perf-node-gather-daemonset-jn4z6\" (UID: \"129ecb30-98f2-42e5-ba80-98e6c4994ae9\") " pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"
Apr 22 20:26:28.028370 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:28.028231    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9s9xq\" (UniqueName: \"kubernetes.io/projected/129ecb30-98f2-42e5-ba80-98e6c4994ae9-kube-api-access-9s9xq\") pod \"perf-node-gather-daemonset-jn4z6\" (UID: \"129ecb30-98f2-42e5-ba80-98e6c4994ae9\") " pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"
Apr 22 20:26:28.028370 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:28.028237    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/129ecb30-98f2-42e5-ba80-98e6c4994ae9-proc\") pod \"perf-node-gather-daemonset-jn4z6\" (UID: \"129ecb30-98f2-42e5-ba80-98e6c4994ae9\") " pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"
Apr 22 20:26:28.028370 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:28.028261    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/129ecb30-98f2-42e5-ba80-98e6c4994ae9-podres\") pod \"perf-node-gather-daemonset-jn4z6\" (UID: \"129ecb30-98f2-42e5-ba80-98e6c4994ae9\") " pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"
Apr 22 20:26:28.028370 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:28.028296    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/129ecb30-98f2-42e5-ba80-98e6c4994ae9-lib-modules\") pod \"perf-node-gather-daemonset-jn4z6\" (UID: \"129ecb30-98f2-42e5-ba80-98e6c4994ae9\") " pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"
Apr 22 20:26:28.028370 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:28.028204    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/129ecb30-98f2-42e5-ba80-98e6c4994ae9-sys\") pod \"perf-node-gather-daemonset-jn4z6\" (UID: \"129ecb30-98f2-42e5-ba80-98e6c4994ae9\") " pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"
Apr 22 20:26:28.028370 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:28.028356    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/129ecb30-98f2-42e5-ba80-98e6c4994ae9-podres\") pod \"perf-node-gather-daemonset-jn4z6\" (UID: \"129ecb30-98f2-42e5-ba80-98e6c4994ae9\") " pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"
Apr 22 20:26:28.035804 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:28.035777    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s9xq\" (UniqueName: \"kubernetes.io/projected/129ecb30-98f2-42e5-ba80-98e6c4994ae9-kube-api-access-9s9xq\") pod \"perf-node-gather-daemonset-jn4z6\" (UID: \"129ecb30-98f2-42e5-ba80-98e6c4994ae9\") " pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"
Apr 22 20:26:28.098167 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:28.098113    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"
Apr 22 20:26:28.224541 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:28.224516    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"]
Apr 22 20:26:28.226528 ip-10-0-135-245 kubenswrapper[2576]: W0422 20:26:28.226494    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod129ecb30_98f2_42e5_ba80_98e6c4994ae9.slice/crio-ecfd4a14f66950d006dbbe9fb8a6e888513b73e5c8ae45d8296dd8fc0db7253f WatchSource:0}: Error finding container ecfd4a14f66950d006dbbe9fb8a6e888513b73e5c8ae45d8296dd8fc0db7253f: Status 404 returned error can't find the container with id ecfd4a14f66950d006dbbe9fb8a6e888513b73e5c8ae45d8296dd8fc0db7253f
Apr 22 20:26:28.228376 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:28.228356    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 22 20:26:28.748723 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:28.748695    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6hc62_8daccaf9-72fb-4f6d-b1b1-245798df3452/dns/0.log"
Apr 22 20:26:28.769425 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:28.769401    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6hc62_8daccaf9-72fb-4f6d-b1b1-245798df3452/kube-rbac-proxy/0.log"
Apr 22 20:26:28.935427 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:28.935396    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rmblf_f12eca63-f2df-4781-97dd-7968beea64ba/dns-node-resolver/0.log"
Apr 22 20:26:29.087298 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:29.087199    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6" event={"ID":"129ecb30-98f2-42e5-ba80-98e6c4994ae9","Type":"ContainerStarted","Data":"c626c0e012ed028e825c3ec0cdfaac9dd4afc5a7c10e375fc72371754694951f"}
Apr 22 20:26:29.087298 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:29.087243    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6" event={"ID":"129ecb30-98f2-42e5-ba80-98e6c4994ae9","Type":"ContainerStarted","Data":"ecfd4a14f66950d006dbbe9fb8a6e888513b73e5c8ae45d8296dd8fc0db7253f"}
Apr 22 20:26:29.087483 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:29.087324    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"
Apr 22 20:26:29.102479 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:29.102438    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6" podStartSLOduration=2.102424903 podStartE2EDuration="2.102424903s" podCreationTimestamp="2026-04-22 20:26:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:26:29.10167338 +0000 UTC m=+1724.201629646" watchObservedRunningTime="2026-04-22 20:26:29.102424903 +0000 UTC m=+1724.202381167"
Apr 22 20:26:29.395010 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:29.394975    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5db49c5564-f22ph_6f66e796-5123-4e93-bfe8-da6796c18fe9/registry/0.log"
Apr 22 20:26:29.460879 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:29.460838    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tn59d_671f3faa-9804-447f-acb1-14a0c42815dd/node-ca/0.log"
Apr 22 20:26:30.311436 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:30.311405    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-55ff986f96-92qlr_a36f74c3-7c15-47c1-baaa-4043e58c120c/discovery/0.log"
Apr 22 20:26:30.333788 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:30.333761    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-7c5447bb76-w9nb5_1cf701c3-b578-4cc8-a6af-4084c426e3ef/istio-proxy/0.log"
Apr 22 20:26:30.786374 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:30.786343    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gdmz7_459bfb9a-58fc-428e-80a8-2813ad5247b8/serve-healthcheck-canary/0.log"
Apr 22 20:26:31.273349 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:31.273293    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4xsbf_a4957aa1-9cf4-499e-af91-57083b2288d6/kube-rbac-proxy/0.log"
Apr 22 20:26:31.294915 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:31.294887    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4xsbf_a4957aa1-9cf4-499e-af91-57083b2288d6/exporter/0.log"
Apr 22 20:26:31.315244 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:31.315224    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4xsbf_a4957aa1-9cf4-499e-af91-57083b2288d6/extractor/0.log"
Apr 22 20:26:34.519653 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:34.519605    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-665c47d676-tslkg_9eaf6ba0-561f-466e-83f3-ad224f3657dd/manager/0.log"
Apr 22 20:26:34.594362 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:34.594329    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-tgxpt_1f1a954a-ca38-4aa2-bb51-2a695fae1578/server/0.log"
Apr 22 20:26:34.790556 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:34.790460    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-2v5hn_9b2bf696-489c-4ee6-8890-43c6b28d60ab/manager/0.log"
Apr 22 20:26:34.812512 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:34.812471    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-nghld_f918f118-912f-463f-a957-a552bbdce85e/s3-init/0.log"
Apr 22 20:26:34.839864 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:34.839839    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-86cc847c5c-swv2m_39cebb9b-69bf-4cb0-a731-69e9bd9851f2/seaweedfs/0.log"
Apr 22 20:26:35.102397 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:35.102317    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7cxtv/perf-node-gather-daemonset-jn4z6"
Apr 22 20:26:41.153299 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:41.153268    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q25vj_dc0bc97c-7cea-48d0-a254-fb23e782e797/kube-multus-additional-cni-plugins/0.log"
Apr 22 20:26:41.175660 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:41.175634    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q25vj_dc0bc97c-7cea-48d0-a254-fb23e782e797/egress-router-binary-copy/0.log"
Apr 22 20:26:41.196012 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:41.195984    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q25vj_dc0bc97c-7cea-48d0-a254-fb23e782e797/cni-plugins/0.log"
Apr 22 20:26:41.216909 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:41.216886    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q25vj_dc0bc97c-7cea-48d0-a254-fb23e782e797/bond-cni-plugin/0.log"
Apr 22 20:26:41.239518 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:41.239483    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q25vj_dc0bc97c-7cea-48d0-a254-fb23e782e797/routeoverride-cni/0.log"
Apr 22 20:26:41.261575 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:41.261548    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q25vj_dc0bc97c-7cea-48d0-a254-fb23e782e797/whereabouts-cni-bincopy/0.log"
Apr 22 20:26:41.281743 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:41.281713    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q25vj_dc0bc97c-7cea-48d0-a254-fb23e782e797/whereabouts-cni/0.log"
Apr 22 20:26:41.342987 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:41.342957    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rzjvv_d5f06d35-71cb-4bf8-af1b-e806af89003e/kube-multus/0.log"
Apr 22 20:26:41.407773 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:41.407695    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lkqdc_7dd397be-1edf-42eb-8b45-a3291c8c3b7f/network-metrics-daemon/0.log"
Apr 22 20:26:41.431346 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:41.431323    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lkqdc_7dd397be-1edf-42eb-8b45-a3291c8c3b7f/kube-rbac-proxy/0.log"
Apr 22 20:26:42.559562 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:42.559526    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rbkp9_ffc78e12-8bf6-4247-b1b7-b3da2c727db5/ovn-controller/0.log"
Apr 22 20:26:42.584659 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:42.584631    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rbkp9_ffc78e12-8bf6-4247-b1b7-b3da2c727db5/ovn-acl-logging/0.log"
Apr 22 20:26:42.601963 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:42.601936    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rbkp9_ffc78e12-8bf6-4247-b1b7-b3da2c727db5/kube-rbac-proxy-node/0.log"
Apr 22 20:26:42.622196 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:42.622168    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rbkp9_ffc78e12-8bf6-4247-b1b7-b3da2c727db5/kube-rbac-proxy-ovn-metrics/0.log"
Apr 22 20:26:42.643438 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:42.643410    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rbkp9_ffc78e12-8bf6-4247-b1b7-b3da2c727db5/northd/0.log"
Apr 22 20:26:42.667518 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:42.667491    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rbkp9_ffc78e12-8bf6-4247-b1b7-b3da2c727db5/nbdb/0.log"
Apr 22 20:26:42.689900 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:42.689871    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rbkp9_ffc78e12-8bf6-4247-b1b7-b3da2c727db5/sbdb/0.log"
Apr 22 20:26:42.806238 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:42.806193    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rbkp9_ffc78e12-8bf6-4247-b1b7-b3da2c727db5/ovnkube-controller/0.log"
Apr 22 20:26:44.209416 ip-10-0-135-245 kubenswrapper[2576]: I0422 20:26:44.209380    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-pq8s7_7f4a64a7-a0cc-426a-9469-5cb2654b3b1d/network-check-target-container/0.log"