Apr 22 19:57:28.464706 ip-10-0-135-215 systemd[1]: Starting Kubernetes Kubelet...
Apr 22 19:57:28.896338 ip-10-0-135-215 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 22 19:57:28.896338 ip-10-0-135-215 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Apr 22 19:57:28.896338 ip-10-0-135-215 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 22 19:57:28.896338 ip-10-0-135-215 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Apr 22 19:57:28.896338 ip-10-0-135-215 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 22 19:57:28.898118 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.897988    2574 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Apr 22 19:57:28.904297 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904282    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 22 19:57:28.904297 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904297    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 22 19:57:28.904362 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904301    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 22 19:57:28.904362 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904306    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 22 19:57:28.904362 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904310    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 22 19:57:28.904362 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904313    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 22 19:57:28.904362 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904316    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 22 19:57:28.904362 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904319    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 22 19:57:28.904362 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904322    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 22 19:57:28.904362 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904328    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 22 19:57:28.904362 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904331    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 22 19:57:28.904362 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904334    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 22 19:57:28.904362 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904337    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 22 19:57:28.904362 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904340    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 22 19:57:28.904362 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904343    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 22 19:57:28.904362 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904345    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 22 19:57:28.904362 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904348    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 22 19:57:28.904362 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904351    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 22 19:57:28.904362 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904353    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 22 19:57:28.904362 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904357    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 22 19:57:28.904362 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904363    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 22 19:57:28.904362 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904370    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 22 19:57:28.904898 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904373    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 22 19:57:28.904898 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904376    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 22 19:57:28.904898 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904379    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 22 19:57:28.904898 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904382    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 22 19:57:28.904898 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904385    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 22 19:57:28.904898 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904387    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 22 19:57:28.904898 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904390    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 22 19:57:28.904898 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904392    2574 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 22 19:57:28.904898 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904395    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 22 19:57:28.904898 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904398    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 22 19:57:28.904898 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904400    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 22 19:57:28.904898 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904405    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 22 19:57:28.904898 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904408    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 22 19:57:28.904898 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904422    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 22 19:57:28.904898 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904425    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 22 19:57:28.904898 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904428    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 22 19:57:28.904898 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904430    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 22 19:57:28.904898 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904433    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 22 19:57:28.904898 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904436    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 22 19:57:28.904898 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904438    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 22 19:57:28.905365 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904441    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 22 19:57:28.905365 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904443    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 22 19:57:28.905365 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904446    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 22 19:57:28.905365 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904448    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 22 19:57:28.905365 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904453    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 22 19:57:28.905365 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904456    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 22 19:57:28.905365 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904459    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 22 19:57:28.905365 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904463    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 22 19:57:28.905365 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904466    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 22 19:57:28.905365 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904469    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 22 19:57:28.905365 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904472    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 22 19:57:28.905365 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904474    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 22 19:57:28.905365 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904477    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 22 19:57:28.905365 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904480    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 22 19:57:28.905365 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904483    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 22 19:57:28.905365 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904486    2574 feature_gate.go:328] unrecognized feature gate: Example
Apr 22 19:57:28.905365 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904488    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 22 19:57:28.905365 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904493    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 22 19:57:28.905365 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904496    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 22 19:57:28.905365 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904499    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 22 19:57:28.905878 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904502    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 22 19:57:28.905878 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904504    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 22 19:57:28.905878 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904507    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 22 19:57:28.905878 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904510    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 22 19:57:28.905878 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904512    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 22 19:57:28.905878 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904515    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 22 19:57:28.905878 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904517    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 22 19:57:28.905878 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904520    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 22 19:57:28.905878 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904523    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 22 19:57:28.905878 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904527    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 22 19:57:28.905878 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904530    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 22 19:57:28.905878 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904533    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 22 19:57:28.905878 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904535    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 22 19:57:28.905878 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904716    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 22 19:57:28.905878 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904724    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 22 19:57:28.905878 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904727    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 22 19:57:28.905878 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904730    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 22 19:57:28.905878 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904733    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 22 19:57:28.905878 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904737    2574 feature_gate.go:328] unrecognized feature gate: Example2
Apr 22 19:57:28.906333 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904745    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 22 19:57:28.906333 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904747    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 22 19:57:28.906333 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904750    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 22 19:57:28.906333 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904753    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 22 19:57:28.906333 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.904756    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 22 19:57:28.906333 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905167    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 22 19:57:28.906333 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905172    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 22 19:57:28.906333 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905176    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 22 19:57:28.906333 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905178    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 22 19:57:28.906333 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905181    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 22 19:57:28.906333 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905183    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 22 19:57:28.906333 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905186    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 22 19:57:28.906333 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905189    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 22 19:57:28.906333 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905191    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 22 19:57:28.906333 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905193    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 22 19:57:28.906333 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905196    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 22 19:57:28.906333 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905198    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 22 19:57:28.906333 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905201    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 22 19:57:28.906333 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905203    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 22 19:57:28.906333 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905207    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 22 19:57:28.906840 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905210    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 22 19:57:28.906840 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905213    2574 feature_gate.go:328] unrecognized feature gate: Example
Apr 22 19:57:28.906840 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905215    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 22 19:57:28.906840 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905218    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 22 19:57:28.906840 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905220    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 22 19:57:28.906840 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905224    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 22 19:57:28.906840 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905228    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 22 19:57:28.906840 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905230    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 22 19:57:28.906840 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905233    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 22 19:57:28.906840 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905235    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 22 19:57:28.906840 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905238    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 22 19:57:28.906840 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905240    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 22 19:57:28.906840 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905242    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 22 19:57:28.906840 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905245    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 22 19:57:28.906840 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905248    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 22 19:57:28.906840 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905250    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 22 19:57:28.906840 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905252    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 22 19:57:28.906840 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905255    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 22 19:57:28.906840 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905257    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 22 19:57:28.906840 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905259    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 22 19:57:28.907321 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905262    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 22 19:57:28.907321 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905264    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 22 19:57:28.907321 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905267    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 22 19:57:28.907321 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905269    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 22 19:57:28.907321 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905271    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 22 19:57:28.907321 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905274    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 22 19:57:28.907321 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905276    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 22 19:57:28.907321 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905278    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 22 19:57:28.907321 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905282    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 22 19:57:28.907321 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905284    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 22 19:57:28.907321 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905287    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 22 19:57:28.907321 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905289    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 22 19:57:28.907321 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905292    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 22 19:57:28.907321 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905294    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 22 19:57:28.907321 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905297    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 22 19:57:28.907321 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905299    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 22 19:57:28.907321 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905301    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 22 19:57:28.907321 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905304    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 22 19:57:28.907321 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905308    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 22 19:57:28.907321 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905312    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 22 19:57:28.907821 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905315    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 22 19:57:28.907821 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905318    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 22 19:57:28.907821 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905320    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 22 19:57:28.907821 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905323    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 22 19:57:28.907821 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905325    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 22 19:57:28.907821 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905327    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 22 19:57:28.907821 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905329    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 22 19:57:28.907821 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905332    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 22 19:57:28.907821 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905334    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 22 19:57:28.907821 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905337    2574 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 22 19:57:28.907821 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905339    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 22 19:57:28.907821 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905342    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 22 19:57:28.907821 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905344    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 22 19:57:28.907821 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905347    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 22 19:57:28.907821 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905349    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 22 19:57:28.907821 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905352    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 22 19:57:28.907821 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905355    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 22 19:57:28.907821 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905357    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 22 19:57:28.907821 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905359    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 22 19:57:28.908269 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905362    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 22 19:57:28.908269 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905365    2574 feature_gate.go:328] unrecognized feature gate: Example2
Apr 22 19:57:28.908269 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905367    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 22 19:57:28.908269 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905370    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 22 19:57:28.908269 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905372    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 22 19:57:28.908269 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905375    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 22 19:57:28.908269 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905377    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 22 19:57:28.908269 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905382    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 22 19:57:28.908269 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905386    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 22 19:57:28.908269 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905389    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 22 19:57:28.908269 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905392    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 22 19:57:28.908269 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.905395    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 22 19:57:28.908269 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906907    2574 flags.go:64] FLAG: --address="0.0.0.0"
Apr 22 19:57:28.908269 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906918    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Apr 22 19:57:28.908269 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906924    2574 flags.go:64] FLAG: --anonymous-auth="true"
Apr 22 19:57:28.908269 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906929    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Apr 22 19:57:28.908269 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906933    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Apr 22 19:57:28.908269 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906937    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Apr 22 19:57:28.908269 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906941    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Apr 22 19:57:28.908269 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906946    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Apr 22 19:57:28.908269 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906949    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906952    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906955    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906958    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906961    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906964    2574 flags.go:64] FLAG: --cgroup-root=""
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906967    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906970    2574 flags.go:64] FLAG: --client-ca-file=""
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906973    2574 flags.go:64] FLAG: --cloud-config=""
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906975    2574 flags.go:64] FLAG: --cloud-provider="external"
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906978    2574 flags.go:64] FLAG: --cluster-dns="[]"
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906983    2574 flags.go:64] FLAG: --cluster-domain=""
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906986    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906988    2574 flags.go:64] FLAG: --config-dir=""
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906991    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906995    2574 flags.go:64] FLAG: --container-log-max-files="5"
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.906999    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907002    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907005    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907009    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907012    2574 flags.go:64] FLAG: --contention-profiling="false"
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907015    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907018    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907021    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907023    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Apr 22 19:57:28.908819 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907027    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907031    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907034    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907037    2574 flags.go:64] FLAG: --enable-load-reader="false"
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907039    2574 flags.go:64] FLAG: --enable-server="true"
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907042    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907046    2574 flags.go:64] FLAG: --event-burst="100"
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907049    2574 flags.go:64] FLAG: --event-qps="50"
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907052    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907055    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907058    2574 flags.go:64] FLAG: --eviction-hard=""
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907062    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907065    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907068    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907071    2574 flags.go:64] FLAG: --eviction-soft=""
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907074    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907077    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907080    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907083    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907086    2574 flags.go:64] FLAG: --fail-cgroupv1="false"
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907088    2574 flags.go:64] FLAG: --fail-swap-on="true"
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907091    2574 flags.go:64] FLAG: --feature-gates=""
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907094    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907097    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907100    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Apr 22 19:57:28.909403 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907103    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907107    2574 flags.go:64] FLAG: --healthz-port="10248"
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907110    2574 flags.go:64] FLAG: --help="false"
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907115    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-135-215.ec2.internal"
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907118    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907121    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907124    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907127    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907131    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907134    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907136    2574 flags.go:64] FLAG: --image-service-endpoint=""
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907139    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907143    2574 flags.go:64] FLAG: --kube-api-burst="100"
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907146    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907149    2574 flags.go:64] FLAG: --kube-api-qps="50"
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907151    2574 flags.go:64] FLAG: --kube-reserved=""
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907154    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907157    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907160    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907164    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907167    2574 flags.go:64] FLAG: --lock-file=""
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907169    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907172    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907175    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Apr 22 19:57:28.910039 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907185    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907188    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907191    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907194    2574 flags.go:64] FLAG: --logging-format="text"
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907196    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907200    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907203    2574 flags.go:64] FLAG: --manifest-url=""
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907205    2574 flags.go:64] FLAG: --manifest-url-header=""
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907210    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907212    2574 flags.go:64] FLAG: --max-open-files="1000000"
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907217    2574 flags.go:64] FLAG: --max-pods="110"
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907220    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907223    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907226    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907229    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907232    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907235    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907238    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907246    2574 flags.go:64] FLAG: --node-status-max-images="50"
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907249    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907252    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907255    2574 flags.go:64] FLAG: --pod-cidr=""
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907258    2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8cfe89231412ff3ee8cb6207fa0be33cad0f08e88c9c0f1e9f7e8c6f14d6715"
Apr 22 19:57:28.910626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907263    2574 flags.go:64] FLAG: --pod-manifest-path=""
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907266    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907269    2574 flags.go:64] FLAG: --pods-per-core="0"
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907272    2574 flags.go:64] FLAG: --port="10250"
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907275    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907279    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06e59e897d34155d1"
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907282    2574 flags.go:64] FLAG: --qos-reserved=""
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907285    2574 flags.go:64] FLAG: --read-only-port="10255"
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907287    2574 flags.go:64] FLAG: --register-node="true"
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907290    2574 flags.go:64] FLAG: --register-schedulable="true"
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907293    2574 flags.go:64] FLAG: --register-with-taints=""
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907297    2574 flags.go:64] FLAG: --registry-burst="10"
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907300    2574 flags.go:64] FLAG: --registry-qps="5"
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907303    2574 flags.go:64] FLAG: --reserved-cpus=""
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907305    2574 flags.go:64] FLAG: --reserved-memory=""
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907309    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907312    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907315    2574 flags.go:64] FLAG: --rotate-certificates="false"
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907317    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907320    2574 flags.go:64] FLAG: --runonce="false"
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907324    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907327    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907330    2574 flags.go:64] FLAG: --seccomp-default="false"
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907333    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907336    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907339    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Apr 22 19:57:28.911184 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907342    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907345    2574 flags.go:64] FLAG: --storage-driver-password="root"
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907349    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907352    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907354    2574 flags.go:64] FLAG: --storage-driver-user="root"
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907357    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907360    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907363    2574 flags.go:64] FLAG: --system-cgroups=""
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907366    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907371    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907374    2574 flags.go:64] FLAG: --tls-cert-file=""
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907377    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907381    2574 flags.go:64] FLAG: --tls-min-version=""
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907384    2574 flags.go:64] FLAG: --tls-private-key-file=""
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907387    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907389    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907392    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907395    2574 flags.go:64] FLAG: --v="2"
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907399    2574 flags.go:64] FLAG: --version="false"
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907403    2574 flags.go:64] FLAG: --vmodule=""
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907407    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.907424    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907516    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907519    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 22 19:57:28.911856 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907523    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 22 19:57:28.912434 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907525    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 22 19:57:28.912434 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907529    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 22 19:57:28.912434 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907532    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 22 19:57:28.912434 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907535    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 22 19:57:28.912434 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907537    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 22 19:57:28.912434 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907541    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 22 19:57:28.912434 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907545    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 22 19:57:28.912434 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907547    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 22 19:57:28.912434 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907550    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 22 19:57:28.912434 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907553    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 22 19:57:28.912434 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907556    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 22 19:57:28.912434 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907558    2574 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 22 19:57:28.912434 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907561    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 22 19:57:28.912434 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907564    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 22 19:57:28.912434 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907568    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 22 19:57:28.912434 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907571    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 22 19:57:28.912434 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907574    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 22 19:57:28.912434 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907577    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 22 19:57:28.912434 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907581    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 22 19:57:28.912922 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907583    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 22 19:57:28.912922 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907586    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 22 19:57:28.912922 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907588    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 22 19:57:28.912922 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907591    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 22 19:57:28.912922 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907593    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 22 19:57:28.912922 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907596    2574 feature_gate.go:328] unrecognized feature gate: Example2
Apr 22 19:57:28.912922 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907599    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 22 19:57:28.912922 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907601    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 22 19:57:28.912922 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907604    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 22 19:57:28.912922 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907606    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 22 19:57:28.912922 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907609    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 22 19:57:28.912922 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907611    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 22 19:57:28.912922 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907614    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 22 19:57:28.912922 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907616    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 22 19:57:28.912922 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907619    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 22 19:57:28.912922 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907622    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 22 19:57:28.912922 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907624    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 22 19:57:28.912922 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907627    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 22 19:57:28.912922 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907629    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 22 19:57:28.912922 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907632    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 22 19:57:28.913423 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907634    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 22 19:57:28.913423 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907637    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 22 19:57:28.913423 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907639    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 22 19:57:28.913423 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907641    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 22 19:57:28.913423 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907647    2574 feature_gate.go:328] unrecognized feature gate: Example
Apr 22 19:57:28.913423 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907649    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 22 19:57:28.913423 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907652    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 22 19:57:28.913423 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907654    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 22 19:57:28.913423 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907657    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 22 19:57:28.913423 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907659    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 22 19:57:28.913423 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907662    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 22 19:57:28.913423 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907665    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 22 19:57:28.913423 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907668    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 22 19:57:28.913423 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907670    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 22 19:57:28.913423 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907673    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 22 19:57:28.913423 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907675    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 22 19:57:28.913423 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907678    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 22 19:57:28.913423 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907680    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 22 19:57:28.913423 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907683    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 22 19:57:28.913879 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907685    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 22 19:57:28.913879 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907687    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 22 19:57:28.913879 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907690    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 22 19:57:28.913879 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907692    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 22 19:57:28.913879 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907695    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 22 19:57:28.913879 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907697    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 22 19:57:28.913879 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907700    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 22 19:57:28.913879 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907702    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 22 19:57:28.913879 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907705    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 22 19:57:28.913879 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907707    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 22 19:57:28.913879 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907710    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 22 19:57:28.913879 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907712    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 22 19:57:28.913879 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907714    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 22 19:57:28.913879 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907717    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 22 19:57:28.913879 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907719    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 22 19:57:28.913879 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907722    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 22 19:57:28.913879 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907724    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 22 19:57:28.913879 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907727    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 22 19:57:28.913879 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907730    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 22 19:57:28.913879 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907733    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 22 19:57:28.914357 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907735    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 22 19:57:28.914357 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907738    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 22 19:57:28.914357 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907740    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 22 19:57:28.914357 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907742    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 22 19:57:28.914357 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.907746    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 22 19:57:28.914357 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.908467    2574 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Apr 22 19:57:28.914950 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.914931    2574 server.go:530] "Kubelet version" kubeletVersion="v1.33.9"
Apr 22 19:57:28.914989 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.914951    2574 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Apr 22 19:57:28.915020 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.914997    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 22 19:57:28.915020 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915002    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 22 19:57:28.915020 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915005    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 22 19:57:28.915020 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915008    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 22 19:57:28.915020 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915011    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 22 19:57:28.915020 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915014    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 22 19:57:28.915020 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915017    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 22 19:57:28.915020 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915019    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 22 19:57:28.915020 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915021    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 22 19:57:28.915020 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915024    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 22 19:57:28.915294 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915027    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 22 19:57:28.915294 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915030    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 22 19:57:28.915294 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915033    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 22 19:57:28.915294 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915035    2574 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 22 19:57:28.915294 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915037    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 22 19:57:28.915294 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915040    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 22 19:57:28.915294 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915043    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 22 19:57:28.915294 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915045    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 22 19:57:28.915294 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915048    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 22 19:57:28.915294 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915051    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 22 19:57:28.915294 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915053    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 22 19:57:28.915294 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915056    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 22 19:57:28.915294 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915058    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 22 19:57:28.915294 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915062    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 22 19:57:28.915294 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915066    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 22 19:57:28.915294 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915069    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 22 19:57:28.915294 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915071    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 22 19:57:28.915294 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915074    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 22 19:57:28.915294 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915077    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 22 19:57:28.915777 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915080    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 22 19:57:28.915777 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915082    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 22 19:57:28.915777 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915085    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 22 19:57:28.915777 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915088    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 22 19:57:28.915777 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915090    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 22 19:57:28.915777 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915093    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 22 19:57:28.915777 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915095    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 22 19:57:28.915777 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915097    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 22 19:57:28.915777 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915100    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 22 19:57:28.915777 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915102    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 22 19:57:28.915777 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915104    2574 feature_gate.go:328] unrecognized feature gate: Example2
Apr 22 19:57:28.915777 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915107    2574 feature_gate.go:328] unrecognized feature gate: Example
Apr 22 19:57:28.915777 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915109    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 22 19:57:28.915777 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915112    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 22 19:57:28.915777 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915115    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 22 19:57:28.915777 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915118    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 22 19:57:28.915777 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915121    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 22 19:57:28.915777 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915124    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 22 19:57:28.915777 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915127    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 22 19:57:28.915777 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915129    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 22 19:57:28.916270 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915132    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 22 19:57:28.916270 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915134    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 22 19:57:28.916270 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915136    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 22 19:57:28.916270 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915139    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 22 19:57:28.916270 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915141    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 22 19:57:28.916270 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915144    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 22 19:57:28.916270 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915146    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 22 19:57:28.916270 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915148    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 22 19:57:28.916270 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915151    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 22 19:57:28.916270 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915153    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 22 19:57:28.916270 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915156    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 22 19:57:28.916270 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915158    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 22 19:57:28.916270 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915161    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 22 19:57:28.916270 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915163    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 22 19:57:28.916270 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915166    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 22 19:57:28.916270 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915168    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 22 19:57:28.916270 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915171    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 22 19:57:28.916270 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915173    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 22 19:57:28.916270 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915176    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 22 19:57:28.916270 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915178    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 22 19:57:28.916768 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915181    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 22 19:57:28.916768 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915184    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 22 19:57:28.916768 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915188    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 22 19:57:28.916768 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915191    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 22 19:57:28.916768 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915194    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 22 19:57:28.916768 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915196    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 22 19:57:28.916768 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915200    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 22 19:57:28.916768 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915203    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 22 19:57:28.916768 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915205    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 22 19:57:28.916768 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915208    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 22 19:57:28.916768 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915210    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 22 19:57:28.916768 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915213    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 22 19:57:28.916768 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915215    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 22 19:57:28.916768 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915218    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 22 19:57:28.916768 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915221    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 22 19:57:28.916768 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915223    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 22 19:57:28.916768 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915225    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 22 19:57:28.917185 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.915231    2574 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Apr 22 19:57:28.917185 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915328    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 22 19:57:28.917185 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915332    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 22 19:57:28.917185 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915335    2574 feature_gate.go:328] unrecognized feature gate: Example2
Apr 22 19:57:28.917185 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915338    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 22 19:57:28.917185 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915340    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 22 19:57:28.917185 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915343    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 22 19:57:28.917185 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915346    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 22 19:57:28.917185 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915348    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 22 19:57:28.917185 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915350    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 22 19:57:28.917185 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915352    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 22 19:57:28.917185 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915355    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 22 19:57:28.917185 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915358    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 22 19:57:28.917185 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915360    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 22 19:57:28.917185 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915363    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 22 19:57:28.917557 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915365    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 22 19:57:28.917557 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915367    2574 feature_gate.go:328] unrecognized feature gate: Example
Apr 22 19:57:28.917557 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915370    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 22 19:57:28.917557 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915372    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 22 19:57:28.917557 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915375    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 22 19:57:28.917557 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915377    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 22 19:57:28.917557 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915379    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 22 19:57:28.917557 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915382    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 22 19:57:28.917557 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915384    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 22 19:57:28.917557 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915387    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 22 19:57:28.917557 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915389    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 22 19:57:28.917557 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915392    2574 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 22 19:57:28.917557 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915394    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 22 19:57:28.917557 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915397    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 22 19:57:28.917557 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915399    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 22 19:57:28.917557 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915401    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 22 19:57:28.917557 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915404    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 22 19:57:28.917557 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915406    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 22 19:57:28.917557 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915408    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 22 19:57:28.917557 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915426    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 22 19:57:28.918029 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915428    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 22 19:57:28.918029 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915431    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 22 19:57:28.918029 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915434    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 22 19:57:28.918029 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915436    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 22 19:57:28.918029 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915439    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 22 19:57:28.918029 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915441    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 22 19:57:28.918029 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915443    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 22 19:57:28.918029 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915448    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 22 19:57:28.918029 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915451    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 22 19:57:28.918029 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915454    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 22 19:57:28.918029 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915457    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 22 19:57:28.918029 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915460    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 22 19:57:28.918029 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915462    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 22 19:57:28.918029 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915464    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 22 19:57:28.918029 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915467    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 22 19:57:28.918029 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915470    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 22 19:57:28.918029 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915472    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 22 19:57:28.918029 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915474    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 22 19:57:28.918029 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915478    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 22 19:57:28.918499 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915480    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 22 19:57:28.918499 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915483    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 22 19:57:28.918499 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915486    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 22 19:57:28.918499 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915488    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 22 19:57:28.918499 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915490    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 22 19:57:28.918499 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915493    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 22 19:57:28.918499 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915495    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 22 19:57:28.918499 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915498    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 22 19:57:28.918499 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915500    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 22 19:57:28.918499 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915503    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 22 19:57:28.918499 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915505    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 22 19:57:28.918499 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915508    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 22 19:57:28.918499 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915510    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 22 19:57:28.918499 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915513    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 22 19:57:28.918499 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915515    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 22 19:57:28.918499 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915518    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 22 19:57:28.918499 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915520    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 22 19:57:28.918499 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915523    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 22 19:57:28.918499 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915525    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 22 19:57:28.918499 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915528    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 22 19:57:28.918994 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915530    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 22 19:57:28.918994 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915532    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 22 19:57:28.918994 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915534    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 22 19:57:28.918994 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915537    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 22 19:57:28.918994 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915540    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 22 19:57:28.918994 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915542    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 22 19:57:28.918994 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915544    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 22 19:57:28.918994 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915548    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 22 19:57:28.918994 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915550    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 22 19:57:28.918994 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915553    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 22 19:57:28.918994 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915555    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 22 19:57:28.918994 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915558    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 22 19:57:28.918994 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:28.915560    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 22 19:57:28.918994 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.915565    2574 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Apr 22 19:57:28.918994 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.916280    2574 server.go:962] "Client rotation is on, will bootstrap in background"
Apr 22 19:57:28.919359 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.918237    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Apr 22 19:57:28.919359 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.919179    2574 server.go:1019] "Starting client certificate rotation"
Apr 22 19:57:28.919359 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.919271    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Apr 22 19:57:28.919359 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.919306    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Apr 22 19:57:28.942645 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.942627    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Apr 22 19:57:28.945862 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.945848    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Apr 22 19:57:28.958950 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.958929    2574 log.go:25] "Validated CRI v1 runtime API"
Apr 22 19:57:28.964109 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.964094    2574 log.go:25] "Validated CRI v1 image API"
Apr 22 19:57:28.968255 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.968238    2574 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Apr 22 19:57:28.969821 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.969806    2574 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Apr 22 19:57:28.970709 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.970689    2574 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a22f4b11-e8e9-411b-a7c3-f6b6df1d33a2:/dev/nvme0n1p3 acffd4bc-dc6b-489a-99c4-b0a398c00470:/dev/nvme0n1p4]
Apr 22 19:57:28.970767 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.970709    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Apr 22 19:57:28.976397 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.976297    2574 manager.go:217] Machine: {Timestamp:2026-04-22 19:57:28.974287984 +0000 UTC m=+0.391951621 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3120327 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25038254028b695af3e1050b49f6b9 SystemUUID:ec250382-5402-8b69-5af3-e1050b49f6b9 BootID:3e7cb521-8d9c-42af-9135-58e73cb4fc46 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1f:6a:98:d8:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1f:6a:98:d8:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:88:07:72:e7:e4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Apr 22 19:57:28.976397 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.976392    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Apr 22 19:57:28.976501 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.976488    2574 manager.go:233] Version: {KernelVersion:5.14.0-570.107.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260414-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Apr 22 19:57:28.977537 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.977518    2574 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Apr 22 19:57:28.977669 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.977541    2574 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-215.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Apr 22 19:57:28.977711 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.977679    2574 topology_manager.go:138] "Creating topology manager with none policy"
Apr 22 19:57:28.977711 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.977688    2574 container_manager_linux.go:306] "Creating device plugin manager"
Apr 22 19:57:28.977711 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.977701    2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Apr 22 19:57:28.978426 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.978403    2574 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Apr 22 19:57:28.979777 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.979767    2574 state_mem.go:36] "Initialized new in-memory state store"
Apr 22 19:57:28.979882 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.979873    2574 server.go:1267] "Using root directory" path="/var/lib/kubelet"
Apr 22 19:57:28.982363 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.982354    2574 kubelet.go:491] "Attempting to sync node with API server"
Apr 22 19:57:28.982451 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.982368    2574 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
Apr 22 19:57:28.982451 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.982379    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Apr 22 19:57:28.982451 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.982388    2574 kubelet.go:397] "Adding apiserver pod source"
Apr 22 19:57:28.982451 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.982397    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Apr 22 19:57:28.983523 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.983512    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Apr 22 19:57:28.983559 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.983531    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Apr 22 19:57:28.986407 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.986392    2574 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.10-2.rhaos4.20.gita4d0894.el9" apiVersion="v1"
Apr 22 19:57:28.987686 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.987674    2574 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Apr 22 19:57:28.989424 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.989403    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Apr 22 19:57:28.989459 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.989439    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Apr 22 19:57:28.989459 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.989448    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Apr 22 19:57:28.989459 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.989454    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Apr 22 19:57:28.989459 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.989459    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Apr 22 19:57:28.989567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.989465    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Apr 22 19:57:28.989567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.989470    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Apr 22 19:57:28.989567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.989476    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Apr 22 19:57:28.989567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.989483    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Apr 22 19:57:28.989567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.989488    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Apr 22 19:57:28.989567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.989497    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Apr 22 19:57:28.989567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.989505    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Apr 22 19:57:28.991075 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.991064    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Apr 22 19:57:28.991108 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.991078    2574 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
Apr 22 19:57:28.994868 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.994854    2574 watchdog_linux.go:99] "Systemd watchdog is not enabled"
Apr 22 19:57:28.994908 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.994888    2574 server.go:1295] "Started kubelet"
Apr 22 19:57:28.995009 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.994984    2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Apr 22 19:57:28.995125 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.995055    2574 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Apr 22 19:57:28.995176 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.995145    2574 server_v1.go:47] "podresources" method="list" useActivePods=true
Apr 22 19:57:28.995870 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.995850    2574 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-215.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Apr 22 19:57:28.995871 ip-10-0-135-215 systemd[1]: Started Kubernetes Kubelet.
Apr 22 19:57:28.996243 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.996228    2574 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Apr 22 19:57:28.996588 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:28.996558    2574 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Apr 22 19:57:28.996588 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:28.996571    2574 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-215.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Apr 22 19:57:28.996727 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:28.996683    2574 server.go:317] "Adding debug handlers to kubelet server"
Apr 22 19:57:29.000960 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.000941    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Apr 22 19:57:29.001464 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.001446    2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Apr 22 19:57:29.002087 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.002074    2574 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
Apr 22 19:57:29.002233 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.002215    2574 factory.go:55] Registering systemd factory
Apr 22 19:57:29.002338 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.002277    2574 factory.go:223] Registration of the systemd container factory successfully
Apr 22 19:57:29.002482 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.002467    2574 volume_manager.go:295] "The desired_state_of_world populator starts"
Apr 22 19:57:29.002482 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.002483    2574 volume_manager.go:297] "Starting Kubelet Volume Manager"
Apr 22 19:57:29.002621 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.002591    2574 reconstruct.go:97] "Volume reconstruction finished"
Apr 22 19:57:29.002621 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.002597    2574 reconciler.go:26] "Reconciler: start to sync state"
Apr 22 19:57:29.006504 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:29.002887    2574 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-215.ec2.internal\" not found"
Apr 22 19:57:29.006672 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:29.002766    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-215.ec2.internal.18a8c615950c080b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-215.ec2.internal,UID:ip-10-0-135-215.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-215.ec2.internal,},FirstTimestamp:2026-04-22 19:57:28.994867211 +0000 UTC m=+0.412530848,LastTimestamp:2026-04-22 19:57:28.994867211 +0000 UTC m=+0.412530848,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-215.ec2.internal,}"
Apr 22 19:57:29.006796 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.006779    2574 factory.go:153] Registering CRI-O factory
Apr 22 19:57:29.006854 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.006802    2574 factory.go:223] Registration of the crio container factory successfully
Apr 22 19:57:29.006908 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.006893    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Apr 22 19:57:29.006973 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.006921    2574 factory.go:103] Registering Raw factory
Apr 22 19:57:29.007127 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:29.007102    2574 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Apr 22 19:57:29.007235 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.007222    2574 manager.go:1196] Started watching for new ooms in manager
Apr 22 19:57:29.008584 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.008565    2574 manager.go:319] Starting recovery of all containers
Apr 22 19:57:29.010035 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.010011    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5l6gf"
Apr 22 19:57:29.012534 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:29.012507    2574 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Apr 22 19:57:29.012853 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:29.012823    2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-215.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Apr 22 19:57:29.017626 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.017596    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5l6gf"
Apr 22 19:57:29.019862 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.019849    2574 manager.go:324] Recovery completed
Apr 22 19:57:29.023559 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.023528    2574 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 22 19:57:29.026024 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.026009    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-215.ec2.internal" event="NodeHasSufficientMemory"
Apr 22 19:57:29.026073 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.026037    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-215.ec2.internal" event="NodeHasNoDiskPressure"
Apr 22 19:57:29.026073 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.026047    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-215.ec2.internal" event="NodeHasSufficientPID"
Apr 22 19:57:29.026579 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.026560    2574 cpu_manager.go:222] "Starting CPU manager" policy="none"
Apr 22 19:57:29.026579 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.026577    2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Apr 22 19:57:29.026686 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.026591    2574 state_mem.go:36] "Initialized new in-memory state store"
Apr 22 19:57:29.028046 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:29.027987    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-215.ec2.internal.18a8c61596e773a3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-215.ec2.internal,UID:ip-10-0-135-215.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-215.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-215.ec2.internal,},FirstTimestamp:2026-04-22 19:57:29.026024355 +0000 UTC m=+0.443687992,LastTimestamp:2026-04-22 19:57:29.026024355 +0000 UTC m=+0.443687992,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-215.ec2.internal,}"
Apr 22 19:57:29.028970 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.028957    2574 policy_none.go:49] "None policy: Start"
Apr 22 19:57:29.029023 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.028974    2574 memory_manager.go:186] "Starting memorymanager" policy="None"
Apr 22 19:57:29.029023 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.028984    2574 state_mem.go:35] "Initializing new in-memory state store"
Apr 22 19:57:29.075152 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.062483    2574 manager.go:341] "Starting Device Plugin manager"
Apr 22 19:57:29.075152 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:29.062509    2574 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Apr 22 19:57:29.075152 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.062521    2574 server.go:85] "Starting device plugin registration server"
Apr 22 19:57:29.075152 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.062746    2574 eviction_manager.go:189] "Eviction manager: starting control loop"
Apr 22 19:57:29.075152 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.062765    2574 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Apr 22 19:57:29.075152 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.062842    2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Apr 22 19:57:29.075152 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.062919    2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Apr 22 19:57:29.075152 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.062928    2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Apr 22 19:57:29.075152 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:29.064323    2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Apr 22 19:57:29.075152 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:29.064357    2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-215.ec2.internal\" not found"
Apr 22 19:57:29.154137 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.154071    2574 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
Apr 22 19:57:29.155370 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.155350    2574 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
Apr 22 19:57:29.155477 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.155389    2574 status_manager.go:230] "Starting to sync pod status with apiserver"
Apr 22 19:57:29.155477 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.155431    2574 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Apr 22 19:57:29.155477 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.155442    2574 kubelet.go:2451] "Starting kubelet main sync loop"
Apr 22 19:57:29.155613 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:29.155484    2574 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Apr 22 19:57:29.158975 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.158952    2574 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 22 19:57:29.163814 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.163800    2574 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 22 19:57:29.164492 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.164478    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-215.ec2.internal" event="NodeHasSufficientMemory"
Apr 22 19:57:29.164590 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.164509    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-215.ec2.internal" event="NodeHasNoDiskPressure"
Apr 22 19:57:29.164590 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.164525    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-215.ec2.internal" event="NodeHasSufficientPID"
Apr 22 19:57:29.164590 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.164554    2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-215.ec2.internal"
Apr 22 19:57:29.173104 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.173090    2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-215.ec2.internal"
Apr 22 19:57:29.173198 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:29.173112    2574 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-215.ec2.internal\": node \"ip-10-0-135-215.ec2.internal\" not found"
Apr 22 19:57:29.188915 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:29.188895    2574 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-215.ec2.internal\" not found"
Apr 22 19:57:29.255755 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.255735    2574 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-215.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-215.ec2.internal"]
Apr 22 19:57:29.255804 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.255793    2574 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 22 19:57:29.256625 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.256610    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-215.ec2.internal" event="NodeHasSufficientMemory"
Apr 22 19:57:29.256679 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.256637    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-215.ec2.internal" event="NodeHasNoDiskPressure"
Apr 22 19:57:29.256679 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.256647    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-215.ec2.internal" event="NodeHasSufficientPID"
Apr 22 19:57:29.257952 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.257941    2574 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 22 19:57:29.258100 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.258087    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-215.ec2.internal"
Apr 22 19:57:29.258138 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.258113    2574 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 22 19:57:29.258699 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.258684    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-215.ec2.internal" event="NodeHasSufficientMemory"
Apr 22 19:57:29.258773 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.258705    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-215.ec2.internal" event="NodeHasSufficientMemory"
Apr 22 19:57:29.258773 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.258716    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-215.ec2.internal" event="NodeHasNoDiskPressure"
Apr 22 19:57:29.258773 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.258727    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-215.ec2.internal" event="NodeHasNoDiskPressure"
Apr 22 19:57:29.258773 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.258742    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-215.ec2.internal" event="NodeHasSufficientPID"
Apr 22 19:57:29.258773 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.258730    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-215.ec2.internal" event="NodeHasSufficientPID"
Apr 22 19:57:29.259830 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.259816    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-215.ec2.internal"
Apr 22 19:57:29.259888 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.259846    2574 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 22 19:57:29.260923 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.260910    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-215.ec2.internal" event="NodeHasSufficientMemory"
Apr 22 19:57:29.260973 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.260937    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-215.ec2.internal" event="NodeHasNoDiskPressure"
Apr 22 19:57:29.260973 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.260948    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-215.ec2.internal" event="NodeHasSufficientPID"
Apr 22 19:57:29.287982 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:29.287964    2574 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-215.ec2.internal\" not found" node="ip-10-0-135-215.ec2.internal"
Apr 22 19:57:29.289455 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:29.289440    2574 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-215.ec2.internal\" not found"
Apr 22 19:57:29.292134 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:29.292119    2574 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-215.ec2.internal\" not found" node="ip-10-0-135-215.ec2.internal"
Apr 22 19:57:29.303623 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.303604    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e47e61ef9462fa2bb1b9971a46826110-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-215.ec2.internal\" (UID: \"e47e61ef9462fa2bb1b9971a46826110\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-215.ec2.internal"
Apr 22 19:57:29.303692 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.303630    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e47e61ef9462fa2bb1b9971a46826110-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-215.ec2.internal\" (UID: \"e47e61ef9462fa2bb1b9971a46826110\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-215.ec2.internal"
Apr 22 19:57:29.303692 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.303647    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2489d845e4aa755919265a93aff51647-config\") pod \"kube-apiserver-proxy-ip-10-0-135-215.ec2.internal\" (UID: \"2489d845e4aa755919265a93aff51647\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-215.ec2.internal"
Apr 22 19:57:29.389843 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:29.389820    2574 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-215.ec2.internal\" not found"
Apr 22 19:57:29.404214 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.404170    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e47e61ef9462fa2bb1b9971a46826110-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-215.ec2.internal\" (UID: \"e47e61ef9462fa2bb1b9971a46826110\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-215.ec2.internal"
Apr 22 19:57:29.404214 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.404195    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e47e61ef9462fa2bb1b9971a46826110-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-215.ec2.internal\" (UID: \"e47e61ef9462fa2bb1b9971a46826110\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-215.ec2.internal"
Apr 22 19:57:29.404214 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.404211    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2489d845e4aa755919265a93aff51647-config\") pod \"kube-apiserver-proxy-ip-10-0-135-215.ec2.internal\" (UID: \"2489d845e4aa755919265a93aff51647\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-215.ec2.internal"
Apr 22 19:57:29.404375 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.404252    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2489d845e4aa755919265a93aff51647-config\") pod \"kube-apiserver-proxy-ip-10-0-135-215.ec2.internal\" (UID: \"2489d845e4aa755919265a93aff51647\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-215.ec2.internal"
Apr 22 19:57:29.404375 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.404267    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e47e61ef9462fa2bb1b9971a46826110-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-215.ec2.internal\" (UID: \"e47e61ef9462fa2bb1b9971a46826110\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-215.ec2.internal"
Apr 22 19:57:29.404375 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.404288    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e47e61ef9462fa2bb1b9971a46826110-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-215.ec2.internal\" (UID: \"e47e61ef9462fa2bb1b9971a46826110\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-215.ec2.internal"
Apr 22 19:57:29.490605 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:29.490579    2574 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-215.ec2.internal\" not found"
Apr 22 19:57:29.590136 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.590095    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-215.ec2.internal"
Apr 22 19:57:29.591162 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:29.591146    2574 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-215.ec2.internal\" not found"
Apr 22 19:57:29.594812 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.594793    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-215.ec2.internal"
Apr 22 19:57:29.691813 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:29.691788    2574 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-215.ec2.internal\" not found"
Apr 22 19:57:29.792372 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:29.792343    2574 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-215.ec2.internal\" not found"
Apr 22 19:57:29.830455 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.830437    2574 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 22 19:57:29.892425 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:29.892385    2574 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-215.ec2.internal\" not found"
Apr 22 19:57:29.918847 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.918819    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Apr 22 19:57:29.919383 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.918935    2574 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Apr 22 19:57:29.919383 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.918974    2574 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Apr 22 19:57:29.986633 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:29.986581    2574 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 22 19:57:30.001379 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:30.001356    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Apr 22 19:57:30.001774 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:30.001761    2574 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-215.ec2.internal"
Apr 22 19:57:30.013348 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:30.013330    2574 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Apr 22 19:57:30.015622 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:30.015608    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Apr 22 19:57:30.017868 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:30.017853    2574 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-215.ec2.internal"
Apr 22 19:57:30.019763 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:30.019733    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-04-21 19:52:29 +0000 UTC" deadline="2028-02-08 13:24:08.609531268 +0000 UTC"
Apr 22 19:57:30.019763 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:30.019762    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15761h26m38.589771744s"
Apr 22 19:57:30.026191 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:30.026177    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Apr 22 19:57:30.030968 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:30.030956    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-782fj"
Apr 22 19:57:30.039805 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:30.039786    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-782fj"
Apr 22 19:57:30.062012 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:30.061995    2574 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 22 19:57:30.204085 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:30.204054    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2489d845e4aa755919265a93aff51647.slice/crio-237fabcf663d5261bbb29909e435435d7bbe019610135891b2dada8aae13fac6 WatchSource:0}: Error finding container 237fabcf663d5261bbb29909e435435d7bbe019610135891b2dada8aae13fac6: Status 404 returned error can't find the container with id 237fabcf663d5261bbb29909e435435d7bbe019610135891b2dada8aae13fac6
Apr 22 19:57:30.205400 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:30.205380    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47e61ef9462fa2bb1b9971a46826110.slice/crio-0a6c8334e8a237749d5d998712a7cb85ecdb9ef4ae0b5ab6fe1bb36d67766759 WatchSource:0}: Error finding container 0a6c8334e8a237749d5d998712a7cb85ecdb9ef4ae0b5ab6fe1bb36d67766759: Status 404 returned error can't find the container with id 0a6c8334e8a237749d5d998712a7cb85ecdb9ef4ae0b5ab6fe1bb36d67766759
Apr 22 19:57:30.210695 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:30.210674    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 22 19:57:30.983886 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:30.983842    2574 apiserver.go:52] "Watching apiserver"
Apr 22 19:57:30.991659 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:30.991440    2574 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Apr 22 19:57:30.991830 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:30.991806    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-vxzgj","openshift-cluster-node-tuning-operator/tuned-wgvqq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-215.ec2.internal","openshift-multus/multus-7ttkd","openshift-multus/multus-additional-cni-plugins-js9v5","openshift-multus/network-metrics-daemon-9cnw2","openshift-network-diagnostics/network-check-target-95q85","openshift-network-operator/iptables-alerter-8ht5v","openshift-ovn-kubernetes/ovnkube-node-zlwx4","kube-system/konnectivity-agent-wmn8m","kube-system/kube-apiserver-proxy-ip-10-0-135-215.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"]
Apr 22 19:57:30.994547 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:30.994526    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vxzgj"
Apr 22 19:57:30.997545 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:30.997285    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Apr 22 19:57:30.997545 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:30.997301    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Apr 22 19:57:30.997545 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:30.997341    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-smzgt\""
Apr 22 19:57:30.997746 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:30.997554    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Apr 22 19:57:30.997891 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:30.997874    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.000119 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.000078    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Apr 22 19:57:31.000237 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.000078    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vmj5p\""
Apr 22 19:57:31.000690 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.000671    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Apr 22 19:57:31.001993 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.001969    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.003360 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.003339    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.004441 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.004407    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:31.004588 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:31.004554    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cnw2" podUID="cdf7849a-206d-4b5a-ad81-e723c58f4426"
Apr 22 19:57:31.004811 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.004787    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Apr 22 19:57:31.004874 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.004795    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Apr 22 19:57:31.004874 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.004848    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Apr 22 19:57:31.005122 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.005027    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Apr 22 19:57:31.005122 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.005092    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rrllm\""
Apr 22 19:57:31.009953 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.006934    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8ht5v"
Apr 22 19:57:31.009953 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.007908    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:31.009953 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:31.007986    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95q85" podUID="635d0f48-b24f-422c-b672-908590754a91"
Apr 22 19:57:31.010610 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.010591    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.011348 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.011326    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wvvk9\""
Apr 22 19:57:31.012050 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.012022    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Apr 22 19:57:31.012191 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.012174    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Apr 22 19:57:31.012298 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.012178    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wmn8m"
Apr 22 19:57:31.013073 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.013051    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-etc-sysctl-conf\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.013173 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.013106    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-run\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.014701 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.014639    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-sys\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.014701 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.014691    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-var-lib-kubelet\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.014868 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.014730    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f755d56-6d83-48c7-be8f-4efe2af27fbc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.014868 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.014772    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a56d6e25-767b-43f8-b8e0-e1dd0b4879fa-host\") pod \"node-ca-vxzgj\" (UID: \"a56d6e25-767b-43f8-b8e0-e1dd0b4879fa\") " pod="openshift-image-registry/node-ca-vxzgj"
Apr 22 19:57:31.014868 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.014804    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-etc-sysconfig\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.015029 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.014898    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-host\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.015029 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.014929    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5063845e-1c48-48a7-bfa3-b31c3d6f8109-tmp\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.015029 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.014961    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lnbd\" (UniqueName: \"kubernetes.io/projected/5063845e-1c48-48a7-bfa3-b31c3d6f8109-kube-api-access-9lnbd\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.016451 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.016429    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Apr 22 19:57:31.016580 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.016554    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Apr 22 19:57:31.016580 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.016572    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wnd2q\""
Apr 22 19:57:31.016727 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.015035    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-cnibin\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.016727 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.016619    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Apr 22 19:57:31.016727 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.016653    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/84c4bd83-8750-4357-8974-fa2ff745d809-cni-binary-copy\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.016727 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.016680    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-host-var-lib-kubelet\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.016727 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.016713    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-host-run-multus-certs\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.017028 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.016740    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f755d56-6d83-48c7-be8f-4efe2af27fbc-cni-binary-copy\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.017028 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.016773    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a56d6e25-767b-43f8-b8e0-e1dd0b4879fa-serviceca\") pod \"node-ca-vxzgj\" (UID: \"a56d6e25-767b-43f8-b8e0-e1dd0b4879fa\") " pod="openshift-image-registry/node-ca-vxzgj"
Apr 22 19:57:31.017028 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.016839    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scjcs\" (UniqueName: \"kubernetes.io/projected/a56d6e25-767b-43f8-b8e0-e1dd0b4879fa-kube-api-access-scjcs\") pod \"node-ca-vxzgj\" (UID: \"a56d6e25-767b-43f8-b8e0-e1dd0b4879fa\") " pod="openshift-image-registry/node-ca-vxzgj"
Apr 22 19:57:31.017028 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.016867    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-etc-sysctl-d\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.017028 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.016893    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rshfs\""
Apr 22 19:57:31.017028 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.016912    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-multus-socket-dir-parent\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.017028 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.016937    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-host-run-netns\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.017028 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.016953    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Apr 22 19:57:31.017028 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.016969    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-host-var-lib-cni-bin\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.017028 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.016992    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-multus-conf-dir\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.017028 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017022    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f755d56-6d83-48c7-be8f-4efe2af27fbc-cnibin\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017047    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7f755d56-6d83-48c7-be8f-4efe2af27fbc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017070    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017074    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-674sn\" (UniqueName: \"kubernetes.io/projected/7f755d56-6d83-48c7-be8f-4efe2af27fbc-kube-api-access-674sn\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017112    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-hostroot\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017119    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017124    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017137    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-etc-systemd\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017161    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npgkw\" (UniqueName: \"kubernetes.io/projected/84c4bd83-8750-4357-8974-fa2ff745d809-kube-api-access-npgkw\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017189    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w97pm\" (UniqueName: \"kubernetes.io/projected/cdf7849a-206d-4b5a-ad81-e723c58f4426-kube-api-access-w97pm\") pod \"network-metrics-daemon-9cnw2\" (UID: \"cdf7849a-206d-4b5a-ad81-e723c58f4426\") " pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017223    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-host-var-lib-cni-multus\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017233    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017248    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-etc-kubernetes\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017279    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-lib-modules\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017304    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5063845e-1c48-48a7-bfa3-b31c3d6f8109-etc-tuned\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017337    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-system-cni-dir\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017346    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017362    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-multus-cni-dir\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017389    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-host-run-k8s-cni-cncf-io\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017399    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-p42gv\""
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017444    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017458    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.017567 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017502    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Apr 22 19:57:31.018273 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017445    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/84c4bd83-8750-4357-8974-fa2ff745d809-multus-daemon-config\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.018273 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017548    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f755d56-6d83-48c7-be8f-4efe2af27fbc-system-cni-dir\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.018273 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017571    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f755d56-6d83-48c7-be8f-4efe2af27fbc-os-release\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.018273 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017587    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7f755d56-6d83-48c7-be8f-4efe2af27fbc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.018273 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017606    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-os-release\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.018273 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017628    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs\") pod \"network-metrics-daemon-9cnw2\" (UID: \"cdf7849a-206d-4b5a-ad81-e723c58f4426\") " pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:31.018273 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017706    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-etc-modprobe-d\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.018273 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.017742    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-etc-kubernetes\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.020680 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.020531    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Apr 22 19:57:31.020750 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.020741    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Apr 22 19:57:31.020973 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.020954    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k7gmp\""
Apr 22 19:57:31.021255 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.021234    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Apr 22 19:57:31.031998 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.031979    2574 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 22 19:57:31.041614 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.041586    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-21 19:52:30 +0000 UTC" deadline="2027-12-24 04:33:30.766236864 +0000 UTC"
Apr 22 19:57:31.041706 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.041614    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14648h35m59.724626419s"
Apr 22 19:57:31.103278 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.103253    2574 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
Apr 22 19:57:31.118425 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118367    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-etc-sysctl-conf\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.118425 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118405    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-run\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.118671 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118449    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-run-ovn\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.118671 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118473    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-os-release\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.118671 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118495    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs\") pod \"network-metrics-daemon-9cnw2\" (UID: \"cdf7849a-206d-4b5a-ad81-e723c58f4426\") " pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:31.118671 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118515    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-etc-kubernetes\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.118671 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118538    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2dca392c-613c-49fb-a976-970fe078c0c4-host-slash\") pod \"iptables-alerter-8ht5v\" (UID: \"2dca392c-613c-49fb-a976-970fe078c0c4\") " pod="openshift-network-operator/iptables-alerter-8ht5v"
Apr 22 19:57:31.118671 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118560    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c9e91d24-9315-418d-98d2-9750bf166aa2-etc-selinux\") pod \"aws-ebs-csi-driver-node-cr45p\" (UID: \"c9e91d24-9315-418d-98d2-9750bf166aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.118671 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118598    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f755d56-6d83-48c7-be8f-4efe2af27fbc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.118671 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118621    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-etc-sysconfig\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.118671 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118644    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5063845e-1c48-48a7-bfa3-b31c3d6f8109-tmp\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.118671 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:31.118659    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:31.118671 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118668    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-host-cni-bin\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.119129 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118691    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/844c05e7-3e07-4c4a-98d5-359d7da599f2-agent-certs\") pod \"konnectivity-agent-wmn8m\" (UID: \"844c05e7-3e07-4c4a-98d5-359d7da599f2\") " pod="kube-system/konnectivity-agent-wmn8m"
Apr 22 19:57:31.119129 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:31.118743    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs podName:cdf7849a-206d-4b5a-ad81-e723c58f4426 nodeName:}" failed. No retries permitted until 2026-04-22 19:57:31.618701522 +0000 UTC m=+3.036365150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs") pod "network-metrics-daemon-9cnw2" (UID: "cdf7849a-206d-4b5a-ad81-e723c58f4426") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:31.119129 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118773    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2dca392c-613c-49fb-a976-970fe078c0c4-iptables-alerter-script\") pod \"iptables-alerter-8ht5v\" (UID: \"2dca392c-613c-49fb-a976-970fe078c0c4\") " pod="openshift-network-operator/iptables-alerter-8ht5v"
Apr 22 19:57:31.119129 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118778    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-etc-kubernetes\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.119129 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118562    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-etc-sysctl-conf\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.119129 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118814    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c9e91d24-9315-418d-98d2-9750bf166aa2-sys-fs\") pod \"aws-ebs-csi-driver-node-cr45p\" (UID: \"c9e91d24-9315-418d-98d2-9750bf166aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.119129 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118843    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-var-lib-openvswitch\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.119129 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118864    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-run\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.119129 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118872    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/84c4bd83-8750-4357-8974-fa2ff745d809-cni-binary-copy\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.119129 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118897    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-host-var-lib-kubelet\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.119129 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118923    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-host-run-multus-certs\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.119129 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118930    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-os-release\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.119129 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118956    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f755d56-6d83-48c7-be8f-4efe2af27fbc-cni-binary-copy\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.119129 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.118979    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-etc-sysctl-d\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.119129 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.119005    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-multus-socket-dir-parent\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.119129 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.119035    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-host-run-netns\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.119129 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.119060    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-multus-conf-dir\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.120002 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.119063    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f755d56-6d83-48c7-be8f-4efe2af27fbc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.120002 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.119085    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7f755d56-6d83-48c7-be8f-4efe2af27fbc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.120002 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.119111    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-674sn\" (UniqueName: \"kubernetes.io/projected/7f755d56-6d83-48c7-be8f-4efe2af27fbc-kube-api-access-674sn\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.120002 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.119115    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-etc-sysconfig\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.120002 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.119141    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d54b3b0c-260b-4004-95d8-2d981031a8a0-env-overrides\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.120002 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.119167    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-hostroot\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.120002 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.119193    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-npgkw\" (UniqueName: \"kubernetes.io/projected/84c4bd83-8750-4357-8974-fa2ff745d809-kube-api-access-npgkw\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.120002 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.119216    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w97pm\" (UniqueName: \"kubernetes.io/projected/cdf7849a-206d-4b5a-ad81-e723c58f4426-kube-api-access-w97pm\") pod \"network-metrics-daemon-9cnw2\" (UID: \"cdf7849a-206d-4b5a-ad81-e723c58f4426\") " pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:31.120002 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.119261    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-systemd-units\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.120002 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.119286    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-host-run-ovn-kubernetes\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.120002 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.119311    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-host-cni-netd\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.120002 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.119333    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d54b3b0c-260b-4004-95d8-2d981031a8a0-ovn-node-metrics-cert\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.120002 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.119362    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d54b3b0c-260b-4004-95d8-2d981031a8a0-ovnkube-script-lib\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.120002 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.119386    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c9e91d24-9315-418d-98d2-9750bf166aa2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cr45p\" (UID: \"c9e91d24-9315-418d-98d2-9750bf166aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.120002 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.119394    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-multus-conf-dir\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.120002 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.119437    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-lib-modules\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.120002 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.119485    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-host-var-lib-kubelet\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.120714 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.120444    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7f755d56-6d83-48c7-be8f-4efe2af27fbc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.120714 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.120463    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/84c4bd83-8750-4357-8974-fa2ff745d809-cni-binary-copy\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.122259 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.120919    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-hostroot\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.122259 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.121012    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-etc-sysctl-d\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.122259 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.121153    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-multus-socket-dir-parent\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.122259 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.119383    2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Apr 22 19:57:31.122259 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.121389    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-host-run-multus-certs\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.122259 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.121468    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-host-run-netns\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.122259 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.121510    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-node-log\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.122259 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.121537    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d54b3b0c-260b-4004-95d8-2d981031a8a0-ovnkube-config\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.122259 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.121553    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-lib-modules\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.122259 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.121562    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fs94\" (UniqueName: \"kubernetes.io/projected/d54b3b0c-260b-4004-95d8-2d981031a8a0-kube-api-access-5fs94\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.122259 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.121890    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c9e91d24-9315-418d-98d2-9750bf166aa2-registration-dir\") pod \"aws-ebs-csi-driver-node-cr45p\" (UID: \"c9e91d24-9315-418d-98d2-9750bf166aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.122259 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.121915    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-multus-cni-dir\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.122259 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.121927    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f755d56-6d83-48c7-be8f-4efe2af27fbc-cni-binary-copy\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.122259 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.121936    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-host-run-k8s-cni-cncf-io\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.122259 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122011    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f755d56-6d83-48c7-be8f-4efe2af27fbc-system-cni-dir\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.122259 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122045    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-sys\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.122259 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122101    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-host-run-k8s-cni-cncf-io\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.122259 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122155    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-var-lib-kubelet\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.123218 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122153    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f755d56-6d83-48c7-be8f-4efe2af27fbc-system-cni-dir\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.123218 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122188    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c9e91d24-9315-418d-98d2-9750bf166aa2-socket-dir\") pod \"aws-ebs-csi-driver-node-cr45p\" (UID: \"c9e91d24-9315-418d-98d2-9750bf166aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.123218 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122237    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-multus-cni-dir\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.123218 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122306    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-run-openvswitch\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.123218 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122349    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-etc-modprobe-d\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.123218 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122385    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9lnbd\" (UniqueName: \"kubernetes.io/projected/5063845e-1c48-48a7-bfa3-b31c3d6f8109-kube-api-access-9lnbd\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.123218 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122391    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-sys\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.123218 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122457    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-host-slash\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.123218 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122485    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-log-socket\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.123218 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122511    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-etc-modprobe-d\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.123218 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122521    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a56d6e25-767b-43f8-b8e0-e1dd0b4879fa-host\") pod \"node-ca-vxzgj\" (UID: \"a56d6e25-767b-43f8-b8e0-e1dd0b4879fa\") " pod="openshift-image-registry/node-ca-vxzgj"
Apr 22 19:57:31.123218 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122554    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-host\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.123218 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122677    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a56d6e25-767b-43f8-b8e0-e1dd0b4879fa-host\") pod \"node-ca-vxzgj\" (UID: \"a56d6e25-767b-43f8-b8e0-e1dd0b4879fa\") " pod="openshift-image-registry/node-ca-vxzgj"
Apr 22 19:57:31.123218 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122800    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-var-lib-kubelet\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.123218 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122834    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c9e91d24-9315-418d-98d2-9750bf166aa2-device-dir\") pod \"aws-ebs-csi-driver-node-cr45p\" (UID: \"c9e91d24-9315-418d-98d2-9750bf166aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.123218 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122853    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-host\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.123218 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122871    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-etc-openvswitch\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.123965 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122905    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-cnibin\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.123965 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.122957    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a56d6e25-767b-43f8-b8e0-e1dd0b4879fa-serviceca\") pod \"node-ca-vxzgj\" (UID: \"a56d6e25-767b-43f8-b8e0-e1dd0b4879fa\") " pod="openshift-image-registry/node-ca-vxzgj"
Apr 22 19:57:31.123965 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.123007    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-cnibin\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.123965 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.123055    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-scjcs\" (UniqueName: \"kubernetes.io/projected/a56d6e25-767b-43f8-b8e0-e1dd0b4879fa-kube-api-access-scjcs\") pod \"node-ca-vxzgj\" (UID: \"a56d6e25-767b-43f8-b8e0-e1dd0b4879fa\") " pod="openshift-image-registry/node-ca-vxzgj"
Apr 22 19:57:31.123965 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.123115    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-host-var-lib-cni-bin\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.123965 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.123508    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f755d56-6d83-48c7-be8f-4efe2af27fbc-cnibin\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.123965 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.123515    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a56d6e25-767b-43f8-b8e0-e1dd0b4879fa-serviceca\") pod \"node-ca-vxzgj\" (UID: \"a56d6e25-767b-43f8-b8e0-e1dd0b4879fa\") " pod="openshift-image-registry/node-ca-vxzgj"
Apr 22 19:57:31.123965 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.123564    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-etc-systemd\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.123965 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.123565    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f755d56-6d83-48c7-be8f-4efe2af27fbc-cnibin\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.123965 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.123604    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/844c05e7-3e07-4c4a-98d5-359d7da599f2-konnectivity-ca\") pod \"konnectivity-agent-wmn8m\" (UID: \"844c05e7-3e07-4c4a-98d5-359d7da599f2\") " pod="kube-system/konnectivity-agent-wmn8m"
Apr 22 19:57:31.123965 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.123622    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5063845e-1c48-48a7-bfa3-b31c3d6f8109-etc-systemd\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.123965 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.123679    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkzvw\" (UniqueName: \"kubernetes.io/projected/2dca392c-613c-49fb-a976-970fe078c0c4-kube-api-access-zkzvw\") pod \"iptables-alerter-8ht5v\" (UID: \"2dca392c-613c-49fb-a976-970fe078c0c4\") " pod="openshift-network-operator/iptables-alerter-8ht5v"
Apr 22 19:57:31.123965 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.123774    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-host-var-lib-cni-bin\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.123965 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.123818    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49nss\" (UniqueName: \"kubernetes.io/projected/c9e91d24-9315-418d-98d2-9750bf166aa2-kube-api-access-49nss\") pod \"aws-ebs-csi-driver-node-cr45p\" (UID: \"c9e91d24-9315-418d-98d2-9750bf166aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.123965 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.123860    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.123965 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.123906    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-host-var-lib-cni-multus\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.124733 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.123989    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-etc-kubernetes\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.124733 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.124019    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5063845e-1c48-48a7-bfa3-b31c3d6f8109-etc-tuned\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.124733 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.124041    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flqgg\" (UniqueName: \"kubernetes.io/projected/635d0f48-b24f-422c-b672-908590754a91-kube-api-access-flqgg\") pod \"network-check-target-95q85\" (UID: \"635d0f48-b24f-422c-b672-908590754a91\") " pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:31.124733 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.124040    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-host-var-lib-cni-multus\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.124733 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.124058    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-host-kubelet\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.124733 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.124073    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-etc-kubernetes\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.124733 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.124204    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-host-run-netns\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.124733 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.124242    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-run-systemd\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.124733 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.124272    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-system-cni-dir\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.124733 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.124331    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/84c4bd83-8750-4357-8974-fa2ff745d809-multus-daemon-config\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.124733 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.124365    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/84c4bd83-8750-4357-8974-fa2ff745d809-system-cni-dir\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.124733 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.124368    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f755d56-6d83-48c7-be8f-4efe2af27fbc-os-release\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.124733 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.124438    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7f755d56-6d83-48c7-be8f-4efe2af27fbc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.124733 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.124448    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f755d56-6d83-48c7-be8f-4efe2af27fbc-os-release\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.125722 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.124834    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/84c4bd83-8750-4357-8974-fa2ff745d809-multus-daemon-config\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.125722 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.125453    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7f755d56-6d83-48c7-be8f-4efe2af27fbc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.127442 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.127315    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5063845e-1c48-48a7-bfa3-b31c3d6f8109-tmp\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.127442 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.127359    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5063845e-1c48-48a7-bfa3-b31c3d6f8109-etc-tuned\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.130000 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.129934    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w97pm\" (UniqueName: \"kubernetes.io/projected/cdf7849a-206d-4b5a-ad81-e723c58f4426-kube-api-access-w97pm\") pod \"network-metrics-daemon-9cnw2\" (UID: \"cdf7849a-206d-4b5a-ad81-e723c58f4426\") " pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:31.130000 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.129939    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-674sn\" (UniqueName: \"kubernetes.io/projected/7f755d56-6d83-48c7-be8f-4efe2af27fbc-kube-api-access-674sn\") pod \"multus-additional-cni-plugins-js9v5\" (UID: \"7f755d56-6d83-48c7-be8f-4efe2af27fbc\") " pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.131696 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.131658    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scjcs\" (UniqueName: \"kubernetes.io/projected/a56d6e25-767b-43f8-b8e0-e1dd0b4879fa-kube-api-access-scjcs\") pod \"node-ca-vxzgj\" (UID: \"a56d6e25-767b-43f8-b8e0-e1dd0b4879fa\") " pod="openshift-image-registry/node-ca-vxzgj"
Apr 22 19:57:31.133307 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.133282    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npgkw\" (UniqueName: \"kubernetes.io/projected/84c4bd83-8750-4357-8974-fa2ff745d809-kube-api-access-npgkw\") pod \"multus-7ttkd\" (UID: \"84c4bd83-8750-4357-8974-fa2ff745d809\") " pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.133560 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.133515    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lnbd\" (UniqueName: \"kubernetes.io/projected/5063845e-1c48-48a7-bfa3-b31c3d6f8109-kube-api-access-9lnbd\") pod \"tuned-wgvqq\" (UID: \"5063845e-1c48-48a7-bfa3-b31c3d6f8109\") " pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.161547 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.161487    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-215.ec2.internal" event={"ID":"2489d845e4aa755919265a93aff51647","Type":"ContainerStarted","Data":"237fabcf663d5261bbb29909e435435d7bbe019610135891b2dada8aae13fac6"}
Apr 22 19:57:31.162536 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.162511    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-215.ec2.internal" event={"ID":"e47e61ef9462fa2bb1b9971a46826110","Type":"ContainerStarted","Data":"0a6c8334e8a237749d5d998712a7cb85ecdb9ef4ae0b5ab6fe1bb36d67766759"}
Apr 22 19:57:31.224864 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.224839    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-host-cni-netd\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.224864 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.224872    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d54b3b0c-260b-4004-95d8-2d981031a8a0-ovn-node-metrics-cert\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.225067 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.224902    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d54b3b0c-260b-4004-95d8-2d981031a8a0-ovnkube-script-lib\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.225067 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.224923    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c9e91d24-9315-418d-98d2-9750bf166aa2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cr45p\" (UID: \"c9e91d24-9315-418d-98d2-9750bf166aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.225067 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.224946    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-node-log\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.225067 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.224970    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-host-cni-netd\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.225067 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.224990    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d54b3b0c-260b-4004-95d8-2d981031a8a0-ovnkube-config\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.225067 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.224994    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-node-log\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.225067 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225035    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5fs94\" (UniqueName: \"kubernetes.io/projected/d54b3b0c-260b-4004-95d8-2d981031a8a0-kube-api-access-5fs94\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.225067 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225064    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c9e91d24-9315-418d-98d2-9750bf166aa2-registration-dir\") pod \"aws-ebs-csi-driver-node-cr45p\" (UID: \"c9e91d24-9315-418d-98d2-9750bf166aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.225434 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225068    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c9e91d24-9315-418d-98d2-9750bf166aa2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cr45p\" (UID: \"c9e91d24-9315-418d-98d2-9750bf166aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.225434 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225093    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c9e91d24-9315-418d-98d2-9750bf166aa2-socket-dir\") pod \"aws-ebs-csi-driver-node-cr45p\" (UID: \"c9e91d24-9315-418d-98d2-9750bf166aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.225434 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225118    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-run-openvswitch\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.225434 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225189    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-host-slash\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.225434 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225215    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-log-socket\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.225434 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225244    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c9e91d24-9315-418d-98d2-9750bf166aa2-device-dir\") pod \"aws-ebs-csi-driver-node-cr45p\" (UID: \"c9e91d24-9315-418d-98d2-9750bf166aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.225434 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225269    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-etc-openvswitch\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.225434 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225283    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-run-openvswitch\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.225434 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225296    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/844c05e7-3e07-4c4a-98d5-359d7da599f2-konnectivity-ca\") pod \"konnectivity-agent-wmn8m\" (UID: \"844c05e7-3e07-4c4a-98d5-359d7da599f2\") " pod="kube-system/konnectivity-agent-wmn8m"
Apr 22 19:57:31.225434 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225322    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zkzvw\" (UniqueName: \"kubernetes.io/projected/2dca392c-613c-49fb-a976-970fe078c0c4-kube-api-access-zkzvw\") pod \"iptables-alerter-8ht5v\" (UID: \"2dca392c-613c-49fb-a976-970fe078c0c4\") " pod="openshift-network-operator/iptables-alerter-8ht5v"
Apr 22 19:57:31.225434 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225337    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c9e91d24-9315-418d-98d2-9750bf166aa2-registration-dir\") pod \"aws-ebs-csi-driver-node-cr45p\" (UID: \"c9e91d24-9315-418d-98d2-9750bf166aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.225434 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225346    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-49nss\" (UniqueName: \"kubernetes.io/projected/c9e91d24-9315-418d-98d2-9750bf166aa2-kube-api-access-49nss\") pod \"aws-ebs-csi-driver-node-cr45p\" (UID: \"c9e91d24-9315-418d-98d2-9750bf166aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.225434 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225369    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.225434 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225397    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-flqgg\" (UniqueName: \"kubernetes.io/projected/635d0f48-b24f-422c-b672-908590754a91-kube-api-access-flqgg\") pod \"network-check-target-95q85\" (UID: \"635d0f48-b24f-422c-b672-908590754a91\") " pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:31.225434 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225401    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-host-slash\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.225434 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225441    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-host-kubelet\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226145 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225447    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c9e91d24-9315-418d-98d2-9750bf166aa2-socket-dir\") pod \"aws-ebs-csi-driver-node-cr45p\" (UID: \"c9e91d24-9315-418d-98d2-9750bf166aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.226145 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225467    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-log-socket\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226145 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225468    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-host-run-netns\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226145 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225513    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-host-run-netns\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226145 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225508    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-run-systemd\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226145 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225547    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-run-systemd\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226145 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225560    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-run-ovn\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226145 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225602    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2dca392c-613c-49fb-a976-970fe078c0c4-host-slash\") pod \"iptables-alerter-8ht5v\" (UID: \"2dca392c-613c-49fb-a976-970fe078c0c4\") " pod="openshift-network-operator/iptables-alerter-8ht5v"
Apr 22 19:57:31.226145 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225606    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-etc-openvswitch\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226145 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225627    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c9e91d24-9315-418d-98d2-9750bf166aa2-etc-selinux\") pod \"aws-ebs-csi-driver-node-cr45p\" (UID: \"c9e91d24-9315-418d-98d2-9750bf166aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.226145 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225657    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-host-cni-bin\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226145 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225690    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/844c05e7-3e07-4c4a-98d5-359d7da599f2-agent-certs\") pod \"konnectivity-agent-wmn8m\" (UID: \"844c05e7-3e07-4c4a-98d5-359d7da599f2\") " pod="kube-system/konnectivity-agent-wmn8m"
Apr 22 19:57:31.226145 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225715    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2dca392c-613c-49fb-a976-970fe078c0c4-iptables-alerter-script\") pod \"iptables-alerter-8ht5v\" (UID: \"2dca392c-613c-49fb-a976-970fe078c0c4\") " pod="openshift-network-operator/iptables-alerter-8ht5v"
Apr 22 19:57:31.226145 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225717    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-host-kubelet\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226145 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225741    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c9e91d24-9315-418d-98d2-9750bf166aa2-sys-fs\") pod \"aws-ebs-csi-driver-node-cr45p\" (UID: \"c9e91d24-9315-418d-98d2-9750bf166aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.226145 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225766    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-var-lib-openvswitch\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226145 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225769    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226895 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225370    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c9e91d24-9315-418d-98d2-9750bf166aa2-device-dir\") pod \"aws-ebs-csi-driver-node-cr45p\" (UID: \"c9e91d24-9315-418d-98d2-9750bf166aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.226895 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225812    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d54b3b0c-260b-4004-95d8-2d981031a8a0-env-overrides\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226895 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225835    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d54b3b0c-260b-4004-95d8-2d981031a8a0-ovnkube-script-lib\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226895 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225841    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-systemd-units\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226895 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225844    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c9e91d24-9315-418d-98d2-9750bf166aa2-etc-selinux\") pod \"aws-ebs-csi-driver-node-cr45p\" (UID: \"c9e91d24-9315-418d-98d2-9750bf166aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.226895 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225865    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-host-run-ovn-kubernetes\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226895 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225926    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c9e91d24-9315-418d-98d2-9750bf166aa2-sys-fs\") pod \"aws-ebs-csi-driver-node-cr45p\" (UID: \"c9e91d24-9315-418d-98d2-9750bf166aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.226895 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225942    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2dca392c-613c-49fb-a976-970fe078c0c4-host-slash\") pod \"iptables-alerter-8ht5v\" (UID: \"2dca392c-613c-49fb-a976-970fe078c0c4\") " pod="openshift-network-operator/iptables-alerter-8ht5v"
Apr 22 19:57:31.226895 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225950    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-systemd-units\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226895 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225977    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-host-cni-bin\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226895 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225991    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-var-lib-openvswitch\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226895 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.225993    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-host-run-ovn-kubernetes\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226895 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.226043    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d54b3b0c-260b-4004-95d8-2d981031a8a0-run-ovn\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226895 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.226302    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d54b3b0c-260b-4004-95d8-2d981031a8a0-ovnkube-config\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226895 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.226358    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d54b3b0c-260b-4004-95d8-2d981031a8a0-env-overrides\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.226895 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.226374    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2dca392c-613c-49fb-a976-970fe078c0c4-iptables-alerter-script\") pod \"iptables-alerter-8ht5v\" (UID: \"2dca392c-613c-49fb-a976-970fe078c0c4\") " pod="openshift-network-operator/iptables-alerter-8ht5v"
Apr 22 19:57:31.226895 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.226481    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/844c05e7-3e07-4c4a-98d5-359d7da599f2-konnectivity-ca\") pod \"konnectivity-agent-wmn8m\" (UID: \"844c05e7-3e07-4c4a-98d5-359d7da599f2\") " pod="kube-system/konnectivity-agent-wmn8m"
Apr 22 19:57:31.228483 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.228453    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d54b3b0c-260b-4004-95d8-2d981031a8a0-ovn-node-metrics-cert\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.228782 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.228758    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/844c05e7-3e07-4c4a-98d5-359d7da599f2-agent-certs\") pod \"konnectivity-agent-wmn8m\" (UID: \"844c05e7-3e07-4c4a-98d5-359d7da599f2\") " pod="kube-system/konnectivity-agent-wmn8m"
Apr 22 19:57:31.232231 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:31.232208    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 22 19:57:31.232231 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:31.232234    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 22 19:57:31.232385 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:31.232248    2574 projected.go:194] Error preparing data for projected volume kube-api-access-flqgg for pod openshift-network-diagnostics/network-check-target-95q85: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:31.232385 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:31.232303    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/635d0f48-b24f-422c-b672-908590754a91-kube-api-access-flqgg podName:635d0f48-b24f-422c-b672-908590754a91 nodeName:}" failed. No retries permitted until 2026-04-22 19:57:31.732285329 +0000 UTC m=+3.149948955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-flqgg" (UniqueName: "kubernetes.io/projected/635d0f48-b24f-422c-b672-908590754a91-kube-api-access-flqgg") pod "network-check-target-95q85" (UID: "635d0f48-b24f-422c-b672-908590754a91") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:31.233790 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.233769    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fs94\" (UniqueName: \"kubernetes.io/projected/d54b3b0c-260b-4004-95d8-2d981031a8a0-kube-api-access-5fs94\") pod \"ovnkube-node-zlwx4\" (UID: \"d54b3b0c-260b-4004-95d8-2d981031a8a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.234192 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.234144    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkzvw\" (UniqueName: \"kubernetes.io/projected/2dca392c-613c-49fb-a976-970fe078c0c4-kube-api-access-zkzvw\") pod \"iptables-alerter-8ht5v\" (UID: \"2dca392c-613c-49fb-a976-970fe078c0c4\") " pod="openshift-network-operator/iptables-alerter-8ht5v"
Apr 22 19:57:31.234805 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.234782    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49nss\" (UniqueName: \"kubernetes.io/projected/c9e91d24-9315-418d-98d2-9750bf166aa2-kube-api-access-49nss\") pod \"aws-ebs-csi-driver-node-cr45p\" (UID: \"c9e91d24-9315-418d-98d2-9750bf166aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.317159 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.317133    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vxzgj"
Apr 22 19:57:31.325778 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.325752    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wgvqq"
Apr 22 19:57:31.333407 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.333389    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7ttkd"
Apr 22 19:57:31.338999 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.338982    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-js9v5"
Apr 22 19:57:31.348877 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.348850    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8ht5v"
Apr 22 19:57:31.356508 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.356489    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:31.366104 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.366084    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wmn8m"
Apr 22 19:57:31.371831 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.371810    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p"
Apr 22 19:57:31.628307 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.628225    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs\") pod \"network-metrics-daemon-9cnw2\" (UID: \"cdf7849a-206d-4b5a-ad81-e723c58f4426\") " pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:31.628482 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:31.628351    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:31.628482 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:31.628405    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs podName:cdf7849a-206d-4b5a-ad81-e723c58f4426 nodeName:}" failed. No retries permitted until 2026-04-22 19:57:32.628392112 +0000 UTC m=+4.046055741 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs") pod "network-metrics-daemon-9cnw2" (UID: "cdf7849a-206d-4b5a-ad81-e723c58f4426") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:31.829061 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:31.829029    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-flqgg\" (UniqueName: \"kubernetes.io/projected/635d0f48-b24f-422c-b672-908590754a91-kube-api-access-flqgg\") pod \"network-check-target-95q85\" (UID: \"635d0f48-b24f-422c-b672-908590754a91\") " pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:31.829237 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:31.829202    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 22 19:57:31.829237 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:31.829223    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 22 19:57:31.829237 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:31.829235    2574 projected.go:194] Error preparing data for projected volume kube-api-access-flqgg for pod openshift-network-diagnostics/network-check-target-95q85: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:31.829390 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:31.829300    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/635d0f48-b24f-422c-b672-908590754a91-kube-api-access-flqgg podName:635d0f48-b24f-422c-b672-908590754a91 nodeName:}" failed. No retries permitted until 2026-04-22 19:57:32.829280924 +0000 UTC m=+4.246944552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-flqgg" (UniqueName: "kubernetes.io/projected/635d0f48-b24f-422c-b672-908590754a91-kube-api-access-flqgg") pod "network-check-target-95q85" (UID: "635d0f48-b24f-422c-b672-908590754a91") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:31.912541 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:31.912257    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dca392c_613c_49fb_a976_970fe078c0c4.slice/crio-e05fb081b8bee11e65cedbdc6d1ce28164c779c6a5fc7b4567f8a8dcb53ba9ef WatchSource:0}: Error finding container e05fb081b8bee11e65cedbdc6d1ce28164c779c6a5fc7b4567f8a8dcb53ba9ef: Status 404 returned error can't find the container with id e05fb081b8bee11e65cedbdc6d1ce28164c779c6a5fc7b4567f8a8dcb53ba9ef
Apr 22 19:57:31.913741 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:31.913707    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c4bd83_8750_4357_8974_fa2ff745d809.slice/crio-2624bfa83c6673ed746db5a2f1010f5f18b4333ed6e617582c156e1f8253de84 WatchSource:0}: Error finding container 2624bfa83c6673ed746db5a2f1010f5f18b4333ed6e617582c156e1f8253de84: Status 404 returned error can't find the container with id 2624bfa83c6673ed746db5a2f1010f5f18b4333ed6e617582c156e1f8253de84
Apr 22 19:57:31.914387 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:31.914370    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5063845e_1c48_48a7_bfa3_b31c3d6f8109.slice/crio-594eca6632f3f68a1cbb9a664d596855034debea674defe771c2f7f4e28a5a10 WatchSource:0}: Error finding container 594eca6632f3f68a1cbb9a664d596855034debea674defe771c2f7f4e28a5a10: Status 404 returned error can't find the container with id 594eca6632f3f68a1cbb9a664d596855034debea674defe771c2f7f4e28a5a10
Apr 22 19:57:31.919267 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:31.919162    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd54b3b0c_260b_4004_95d8_2d981031a8a0.slice/crio-aa9f588a92bba2233d356c9de46ae7a6b4ccf7bc33ff274ab2b813c489c1192d WatchSource:0}: Error finding container aa9f588a92bba2233d356c9de46ae7a6b4ccf7bc33ff274ab2b813c489c1192d: Status 404 returned error can't find the container with id aa9f588a92bba2233d356c9de46ae7a6b4ccf7bc33ff274ab2b813c489c1192d
Apr 22 19:57:31.920012 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:31.919994    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e91d24_9315_418d_98d2_9750bf166aa2.slice/crio-2d042bc8b70afb0fe742c9f0faf02e6325b8ad7c1067229f446e9704bac07c28 WatchSource:0}: Error finding container 2d042bc8b70afb0fe742c9f0faf02e6325b8ad7c1067229f446e9704bac07c28: Status 404 returned error can't find the container with id 2d042bc8b70afb0fe742c9f0faf02e6325b8ad7c1067229f446e9704bac07c28
Apr 22 19:57:31.921763 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:31.921728    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f755d56_6d83_48c7_be8f_4efe2af27fbc.slice/crio-fd0563f5dac7f27a52ee731e683cddbe9a67903db903be1a3bf8aa2d37c613ae WatchSource:0}: Error finding container fd0563f5dac7f27a52ee731e683cddbe9a67903db903be1a3bf8aa2d37c613ae: Status 404 returned error can't find the container with id fd0563f5dac7f27a52ee731e683cddbe9a67903db903be1a3bf8aa2d37c613ae
Apr 22 19:57:31.922834 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:31.922807    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844c05e7_3e07_4c4a_98d5_359d7da599f2.slice/crio-f3e6c1ea03a77ce8926b844d39847fee00a4e388067f036c81b6116b87c7b482 WatchSource:0}: Error finding container f3e6c1ea03a77ce8926b844d39847fee00a4e388067f036c81b6116b87c7b482: Status 404 returned error can't find the container with id f3e6c1ea03a77ce8926b844d39847fee00a4e388067f036c81b6116b87c7b482
Apr 22 19:57:31.924500 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:31.924478    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda56d6e25_767b_43f8_b8e0_e1dd0b4879fa.slice/crio-554eac7be5d40acd29fa3efe99edcaa71b942b63e89c94629da19f9e6082baae WatchSource:0}: Error finding container 554eac7be5d40acd29fa3efe99edcaa71b942b63e89c94629da19f9e6082baae: Status 404 returned error can't find the container with id 554eac7be5d40acd29fa3efe99edcaa71b942b63e89c94629da19f9e6082baae
Apr 22 19:57:32.042172 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:32.042142    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-21 19:52:30 +0000 UTC" deadline="2028-01-05 14:30:36.106356076 +0000 UTC"
Apr 22 19:57:32.042172 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:32.042167    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14946h33m4.064191447s"
Apr 22 19:57:32.164385 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:32.164287    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js9v5" event={"ID":"7f755d56-6d83-48c7-be8f-4efe2af27fbc","Type":"ContainerStarted","Data":"fd0563f5dac7f27a52ee731e683cddbe9a67903db903be1a3bf8aa2d37c613ae"}
Apr 22 19:57:32.165175 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:32.165149    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wgvqq" event={"ID":"5063845e-1c48-48a7-bfa3-b31c3d6f8109","Type":"ContainerStarted","Data":"594eca6632f3f68a1cbb9a664d596855034debea674defe771c2f7f4e28a5a10"}
Apr 22 19:57:32.166048 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:32.166030    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7ttkd" event={"ID":"84c4bd83-8750-4357-8974-fa2ff745d809","Type":"ContainerStarted","Data":"2624bfa83c6673ed746db5a2f1010f5f18b4333ed6e617582c156e1f8253de84"}
Apr 22 19:57:32.167476 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:32.167449    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-215.ec2.internal" event={"ID":"2489d845e4aa755919265a93aff51647","Type":"ContainerStarted","Data":"0584305f0317bd91987d5e10d20cc57c649257dce5738cd767fbb21d9b00fbb5"}
Apr 22 19:57:32.168354 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:32.168325    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vxzgj" event={"ID":"a56d6e25-767b-43f8-b8e0-e1dd0b4879fa","Type":"ContainerStarted","Data":"554eac7be5d40acd29fa3efe99edcaa71b942b63e89c94629da19f9e6082baae"}
Apr 22 19:57:32.169213 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:32.169187    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wmn8m" event={"ID":"844c05e7-3e07-4c4a-98d5-359d7da599f2","Type":"ContainerStarted","Data":"f3e6c1ea03a77ce8926b844d39847fee00a4e388067f036c81b6116b87c7b482"}
Apr 22 19:57:32.170239 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:32.170212    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p" event={"ID":"c9e91d24-9315-418d-98d2-9750bf166aa2","Type":"ContainerStarted","Data":"2d042bc8b70afb0fe742c9f0faf02e6325b8ad7c1067229f446e9704bac07c28"}
Apr 22 19:57:32.171695 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:32.171678    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4" event={"ID":"d54b3b0c-260b-4004-95d8-2d981031a8a0","Type":"ContainerStarted","Data":"aa9f588a92bba2233d356c9de46ae7a6b4ccf7bc33ff274ab2b813c489c1192d"}
Apr 22 19:57:32.172633 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:32.172614    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8ht5v" event={"ID":"2dca392c-613c-49fb-a976-970fe078c0c4","Type":"ContainerStarted","Data":"e05fb081b8bee11e65cedbdc6d1ce28164c779c6a5fc7b4567f8a8dcb53ba9ef"}
Apr 22 19:57:32.180369 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:32.180328    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-215.ec2.internal" podStartSLOduration=2.1803166 podStartE2EDuration="2.1803166s" podCreationTimestamp="2026-04-22 19:57:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 19:57:32.179873436 +0000 UTC m=+3.597537076" watchObservedRunningTime="2026-04-22 19:57:32.1803166 +0000 UTC m=+3.597980245"
Apr 22 19:57:32.629468 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:32.629401    2574 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 22 19:57:32.639013 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:32.638985    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs\") pod \"network-metrics-daemon-9cnw2\" (UID: \"cdf7849a-206d-4b5a-ad81-e723c58f4426\") " pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:32.639141 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:32.639124    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:32.639210 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:32.639194    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs podName:cdf7849a-206d-4b5a-ad81-e723c58f4426 nodeName:}" failed. No retries permitted until 2026-04-22 19:57:34.639173617 +0000 UTC m=+6.056837242 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs") pod "network-metrics-daemon-9cnw2" (UID: "cdf7849a-206d-4b5a-ad81-e723c58f4426") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:32.845448 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:32.843000    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-flqgg\" (UniqueName: \"kubernetes.io/projected/635d0f48-b24f-422c-b672-908590754a91-kube-api-access-flqgg\") pod \"network-check-target-95q85\" (UID: \"635d0f48-b24f-422c-b672-908590754a91\") " pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:32.845448 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:32.843178    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 22 19:57:32.845448 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:32.843200    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 22 19:57:32.845448 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:32.843212    2574 projected.go:194] Error preparing data for projected volume kube-api-access-flqgg for pod openshift-network-diagnostics/network-check-target-95q85: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:32.845448 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:32.843272    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/635d0f48-b24f-422c-b672-908590754a91-kube-api-access-flqgg podName:635d0f48-b24f-422c-b672-908590754a91 nodeName:}" failed. No retries permitted until 2026-04-22 19:57:34.843252851 +0000 UTC m=+6.260916479 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-flqgg" (UniqueName: "kubernetes.io/projected/635d0f48-b24f-422c-b672-908590754a91-kube-api-access-flqgg") pod "network-check-target-95q85" (UID: "635d0f48-b24f-422c-b672-908590754a91") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:33.141248 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:33.140433    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-89zh2"]
Apr 22 19:57:33.142977 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:33.142510    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:33.142977 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:33.142582    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zh2" podUID="bcfcba31-705d-4865-bba0-528576e44d02"
Apr 22 19:57:33.159246 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:33.158497    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:33.159246 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:33.158589    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95q85" podUID="635d0f48-b24f-422c-b672-908590754a91"
Apr 22 19:57:33.159246 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:33.158919    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:33.159246 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:33.159025    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cnw2" podUID="cdf7849a-206d-4b5a-ad81-e723c58f4426"
Apr 22 19:57:33.199436 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:33.199157    2574 generic.go:358] "Generic (PLEG): container finished" podID="e47e61ef9462fa2bb1b9971a46826110" containerID="2c3142ef8d73357a665eb41eaf3ce88dc9a20a977e1f511e8032d4004f638454" exitCode=0
Apr 22 19:57:33.199436 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:33.199251    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-215.ec2.internal" event={"ID":"e47e61ef9462fa2bb1b9971a46826110","Type":"ContainerDied","Data":"2c3142ef8d73357a665eb41eaf3ce88dc9a20a977e1f511e8032d4004f638454"}
Apr 22 19:57:33.245792 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:33.245760    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bcfcba31-705d-4865-bba0-528576e44d02-dbus\") pod \"global-pull-secret-syncer-89zh2\" (UID: \"bcfcba31-705d-4865-bba0-528576e44d02\") " pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:33.245906 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:33.245808    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcfcba31-705d-4865-bba0-528576e44d02-original-pull-secret\") pod \"global-pull-secret-syncer-89zh2\" (UID: \"bcfcba31-705d-4865-bba0-528576e44d02\") " pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:33.245906 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:33.245847    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bcfcba31-705d-4865-bba0-528576e44d02-kubelet-config\") pod \"global-pull-secret-syncer-89zh2\" (UID: \"bcfcba31-705d-4865-bba0-528576e44d02\") " pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:33.346338 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:33.346302    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bcfcba31-705d-4865-bba0-528576e44d02-kubelet-config\") pod \"global-pull-secret-syncer-89zh2\" (UID: \"bcfcba31-705d-4865-bba0-528576e44d02\") " pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:33.346519 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:33.346385    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bcfcba31-705d-4865-bba0-528576e44d02-dbus\") pod \"global-pull-secret-syncer-89zh2\" (UID: \"bcfcba31-705d-4865-bba0-528576e44d02\") " pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:33.346519 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:33.346435    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcfcba31-705d-4865-bba0-528576e44d02-original-pull-secret\") pod \"global-pull-secret-syncer-89zh2\" (UID: \"bcfcba31-705d-4865-bba0-528576e44d02\") " pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:33.346619 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:33.346555    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Apr 22 19:57:33.346619 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:33.346616    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcfcba31-705d-4865-bba0-528576e44d02-original-pull-secret podName:bcfcba31-705d-4865-bba0-528576e44d02 nodeName:}" failed. No retries permitted until 2026-04-22 19:57:33.846598118 +0000 UTC m=+5.264261749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bcfcba31-705d-4865-bba0-528576e44d02-original-pull-secret") pod "global-pull-secret-syncer-89zh2" (UID: "bcfcba31-705d-4865-bba0-528576e44d02") : object "kube-system"/"original-pull-secret" not registered
Apr 22 19:57:33.346729 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:33.346679    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bcfcba31-705d-4865-bba0-528576e44d02-kubelet-config\") pod \"global-pull-secret-syncer-89zh2\" (UID: \"bcfcba31-705d-4865-bba0-528576e44d02\") " pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:33.346844 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:33.346826    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bcfcba31-705d-4865-bba0-528576e44d02-dbus\") pod \"global-pull-secret-syncer-89zh2\" (UID: \"bcfcba31-705d-4865-bba0-528576e44d02\") " pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:33.850742 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:33.850660    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcfcba31-705d-4865-bba0-528576e44d02-original-pull-secret\") pod \"global-pull-secret-syncer-89zh2\" (UID: \"bcfcba31-705d-4865-bba0-528576e44d02\") " pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:33.850915 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:33.850870    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Apr 22 19:57:33.850980 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:33.850929    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcfcba31-705d-4865-bba0-528576e44d02-original-pull-secret podName:bcfcba31-705d-4865-bba0-528576e44d02 nodeName:}" failed. No retries permitted until 2026-04-22 19:57:34.850912956 +0000 UTC m=+6.268576610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bcfcba31-705d-4865-bba0-528576e44d02-original-pull-secret") pod "global-pull-secret-syncer-89zh2" (UID: "bcfcba31-705d-4865-bba0-528576e44d02") : object "kube-system"/"original-pull-secret" not registered
Apr 22 19:57:34.208552 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:34.207868    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-215.ec2.internal" event={"ID":"e47e61ef9462fa2bb1b9971a46826110","Type":"ContainerStarted","Data":"dcd979f60584ac3f9ad67d3c4460c9af32b63bab8fad876702b63425a02e9175"}
Apr 22 19:57:34.658912 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:34.658827    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs\") pod \"network-metrics-daemon-9cnw2\" (UID: \"cdf7849a-206d-4b5a-ad81-e723c58f4426\") " pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:34.659067 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:34.658995    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:34.659067 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:34.659051    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs podName:cdf7849a-206d-4b5a-ad81-e723c58f4426 nodeName:}" failed. No retries permitted until 2026-04-22 19:57:38.659034837 +0000 UTC m=+10.076698463 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs") pod "network-metrics-daemon-9cnw2" (UID: "cdf7849a-206d-4b5a-ad81-e723c58f4426") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:34.862037 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:34.861439    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-flqgg\" (UniqueName: \"kubernetes.io/projected/635d0f48-b24f-422c-b672-908590754a91-kube-api-access-flqgg\") pod \"network-check-target-95q85\" (UID: \"635d0f48-b24f-422c-b672-908590754a91\") " pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:34.862037 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:34.861542    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcfcba31-705d-4865-bba0-528576e44d02-original-pull-secret\") pod \"global-pull-secret-syncer-89zh2\" (UID: \"bcfcba31-705d-4865-bba0-528576e44d02\") " pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:34.862037 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:34.861621    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 22 19:57:34.862037 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:34.861624    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Apr 22 19:57:34.862037 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:34.861640    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 22 19:57:34.862037 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:34.861650    2574 projected.go:194] Error preparing data for projected volume kube-api-access-flqgg for pod openshift-network-diagnostics/network-check-target-95q85: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:34.862037 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:34.861681    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcfcba31-705d-4865-bba0-528576e44d02-original-pull-secret podName:bcfcba31-705d-4865-bba0-528576e44d02 nodeName:}" failed. No retries permitted until 2026-04-22 19:57:36.861664397 +0000 UTC m=+8.279328038 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bcfcba31-705d-4865-bba0-528576e44d02-original-pull-secret") pod "global-pull-secret-syncer-89zh2" (UID: "bcfcba31-705d-4865-bba0-528576e44d02") : object "kube-system"/"original-pull-secret" not registered
Apr 22 19:57:34.862037 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:34.861701    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/635d0f48-b24f-422c-b672-908590754a91-kube-api-access-flqgg podName:635d0f48-b24f-422c-b672-908590754a91 nodeName:}" failed. No retries permitted until 2026-04-22 19:57:38.861689151 +0000 UTC m=+10.279352778 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-flqgg" (UniqueName: "kubernetes.io/projected/635d0f48-b24f-422c-b672-908590754a91-kube-api-access-flqgg") pod "network-check-target-95q85" (UID: "635d0f48-b24f-422c-b672-908590754a91") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:35.156533 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:35.156506    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:35.156712 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:35.156601    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zh2" podUID="bcfcba31-705d-4865-bba0-528576e44d02"
Apr 22 19:57:35.157003 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:35.156899    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:35.157003 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:35.156965    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95q85" podUID="635d0f48-b24f-422c-b672-908590754a91"
Apr 22 19:57:35.157292 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:35.157193    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:35.157292 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:35.157261    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cnw2" podUID="cdf7849a-206d-4b5a-ad81-e723c58f4426"
Apr 22 19:57:36.876663 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:36.876613    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcfcba31-705d-4865-bba0-528576e44d02-original-pull-secret\") pod \"global-pull-secret-syncer-89zh2\" (UID: \"bcfcba31-705d-4865-bba0-528576e44d02\") " pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:36.877133 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:36.876772    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Apr 22 19:57:36.877133 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:36.876832    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcfcba31-705d-4865-bba0-528576e44d02-original-pull-secret podName:bcfcba31-705d-4865-bba0-528576e44d02 nodeName:}" failed. No retries permitted until 2026-04-22 19:57:40.876814977 +0000 UTC m=+12.294478605 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bcfcba31-705d-4865-bba0-528576e44d02-original-pull-secret") pod "global-pull-secret-syncer-89zh2" (UID: "bcfcba31-705d-4865-bba0-528576e44d02") : object "kube-system"/"original-pull-secret" not registered
Apr 22 19:57:37.157665 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:37.156154    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:37.157665 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:37.156293    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cnw2" podUID="cdf7849a-206d-4b5a-ad81-e723c58f4426"
Apr 22 19:57:37.157665 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:37.156698    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:37.157665 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:37.156787    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zh2" podUID="bcfcba31-705d-4865-bba0-528576e44d02"
Apr 22 19:57:37.157665 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:37.157470    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:37.157665 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:37.157557    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95q85" podUID="635d0f48-b24f-422c-b672-908590754a91"
Apr 22 19:57:38.690238 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:38.690195    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs\") pod \"network-metrics-daemon-9cnw2\" (UID: \"cdf7849a-206d-4b5a-ad81-e723c58f4426\") " pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:38.690697 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:38.690366    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:38.690697 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:38.690450    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs podName:cdf7849a-206d-4b5a-ad81-e723c58f4426 nodeName:}" failed. No retries permitted until 2026-04-22 19:57:46.690430649 +0000 UTC m=+18.108094289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs") pod "network-metrics-daemon-9cnw2" (UID: "cdf7849a-206d-4b5a-ad81-e723c58f4426") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:38.891666 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:38.891621    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-flqgg\" (UniqueName: \"kubernetes.io/projected/635d0f48-b24f-422c-b672-908590754a91-kube-api-access-flqgg\") pod \"network-check-target-95q85\" (UID: \"635d0f48-b24f-422c-b672-908590754a91\") " pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:38.891890 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:38.891856    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 22 19:57:38.891890 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:38.891883    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 22 19:57:38.891963 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:38.891898    2574 projected.go:194] Error preparing data for projected volume kube-api-access-flqgg for pod openshift-network-diagnostics/network-check-target-95q85: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:38.891963 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:38.891959    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/635d0f48-b24f-422c-b672-908590754a91-kube-api-access-flqgg podName:635d0f48-b24f-422c-b672-908590754a91 nodeName:}" failed. No retries permitted until 2026-04-22 19:57:46.891939525 +0000 UTC m=+18.309603154 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-flqgg" (UniqueName: "kubernetes.io/projected/635d0f48-b24f-422c-b672-908590754a91-kube-api-access-flqgg") pod "network-check-target-95q85" (UID: "635d0f48-b24f-422c-b672-908590754a91") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:39.156509 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:39.156427    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:39.156662 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:39.156551    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95q85" podUID="635d0f48-b24f-422c-b672-908590754a91"
Apr 22 19:57:39.158207 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:39.156922    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:39.158207 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:39.157026    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cnw2" podUID="cdf7849a-206d-4b5a-ad81-e723c58f4426"
Apr 22 19:57:39.158207 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:39.158087    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:39.158207 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:39.158173    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zh2" podUID="bcfcba31-705d-4865-bba0-528576e44d02"
Apr 22 19:57:40.907838 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:40.907256    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcfcba31-705d-4865-bba0-528576e44d02-original-pull-secret\") pod \"global-pull-secret-syncer-89zh2\" (UID: \"bcfcba31-705d-4865-bba0-528576e44d02\") " pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:40.907838 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:40.907399    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Apr 22 19:57:40.907838 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:40.907480    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcfcba31-705d-4865-bba0-528576e44d02-original-pull-secret podName:bcfcba31-705d-4865-bba0-528576e44d02 nodeName:}" failed. No retries permitted until 2026-04-22 19:57:48.90746272 +0000 UTC m=+20.325126346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bcfcba31-705d-4865-bba0-528576e44d02-original-pull-secret") pod "global-pull-secret-syncer-89zh2" (UID: "bcfcba31-705d-4865-bba0-528576e44d02") : object "kube-system"/"original-pull-secret" not registered
Apr 22 19:57:41.155960 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:41.155885    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:41.156117 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:41.155885    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:41.156117 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:41.155891    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:41.156117 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:41.156091    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zh2" podUID="bcfcba31-705d-4865-bba0-528576e44d02"
Apr 22 19:57:41.156278 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:41.155986    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95q85" podUID="635d0f48-b24f-422c-b672-908590754a91"
Apr 22 19:57:41.156278 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:41.156185    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cnw2" podUID="cdf7849a-206d-4b5a-ad81-e723c58f4426"
Apr 22 19:57:42.284556 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:42.284508    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-215.ec2.internal" podStartSLOduration=12.284491669 podStartE2EDuration="12.284491669s" podCreationTimestamp="2026-04-22 19:57:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 19:57:34.222592023 +0000 UTC m=+5.640255672" watchObservedRunningTime="2026-04-22 19:57:42.284491669 +0000 UTC m=+13.702155314"
Apr 22 19:57:42.284948 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:42.284817    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-sz29m"]
Apr 22 19:57:42.318106 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:42.318079    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sz29m"
Apr 22 19:57:42.320774 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:42.320746    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7782d\""
Apr 22 19:57:42.320887 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:42.320792    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Apr 22 19:57:42.321362 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:42.321212    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Apr 22 19:57:42.417238 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:42.417190    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4edbe2f-e978-4101-8d32-e35e7e5e740e-hosts-file\") pod \"node-resolver-sz29m\" (UID: \"c4edbe2f-e978-4101-8d32-e35e7e5e740e\") " pod="openshift-dns/node-resolver-sz29m"
Apr 22 19:57:42.417395 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:42.417248    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4edbe2f-e978-4101-8d32-e35e7e5e740e-tmp-dir\") pod \"node-resolver-sz29m\" (UID: \"c4edbe2f-e978-4101-8d32-e35e7e5e740e\") " pod="openshift-dns/node-resolver-sz29m"
Apr 22 19:57:42.417395 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:42.417326    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v5k6\" (UniqueName: \"kubernetes.io/projected/c4edbe2f-e978-4101-8d32-e35e7e5e740e-kube-api-access-2v5k6\") pod \"node-resolver-sz29m\" (UID: \"c4edbe2f-e978-4101-8d32-e35e7e5e740e\") " pod="openshift-dns/node-resolver-sz29m"
Apr 22 19:57:42.517807 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:42.517775    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4edbe2f-e978-4101-8d32-e35e7e5e740e-hosts-file\") pod \"node-resolver-sz29m\" (UID: \"c4edbe2f-e978-4101-8d32-e35e7e5e740e\") " pod="openshift-dns/node-resolver-sz29m"
Apr 22 19:57:42.517968 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:42.517832    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4edbe2f-e978-4101-8d32-e35e7e5e740e-tmp-dir\") pod \"node-resolver-sz29m\" (UID: \"c4edbe2f-e978-4101-8d32-e35e7e5e740e\") " pod="openshift-dns/node-resolver-sz29m"
Apr 22 19:57:42.517968 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:42.517899    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2v5k6\" (UniqueName: \"kubernetes.io/projected/c4edbe2f-e978-4101-8d32-e35e7e5e740e-kube-api-access-2v5k6\") pod \"node-resolver-sz29m\" (UID: \"c4edbe2f-e978-4101-8d32-e35e7e5e740e\") " pod="openshift-dns/node-resolver-sz29m"
Apr 22 19:57:42.517968 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:42.517912    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4edbe2f-e978-4101-8d32-e35e7e5e740e-hosts-file\") pod \"node-resolver-sz29m\" (UID: \"c4edbe2f-e978-4101-8d32-e35e7e5e740e\") " pod="openshift-dns/node-resolver-sz29m"
Apr 22 19:57:42.518238 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:42.518220    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4edbe2f-e978-4101-8d32-e35e7e5e740e-tmp-dir\") pod \"node-resolver-sz29m\" (UID: \"c4edbe2f-e978-4101-8d32-e35e7e5e740e\") " pod="openshift-dns/node-resolver-sz29m"
Apr 22 19:57:42.527239 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:42.527214    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v5k6\" (UniqueName: \"kubernetes.io/projected/c4edbe2f-e978-4101-8d32-e35e7e5e740e-kube-api-access-2v5k6\") pod \"node-resolver-sz29m\" (UID: \"c4edbe2f-e978-4101-8d32-e35e7e5e740e\") " pod="openshift-dns/node-resolver-sz29m"
Apr 22 19:57:42.628805 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:42.628728    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sz29m"
Apr 22 19:57:43.156377 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:43.156338    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:43.156554 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:43.156338    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:43.156771 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:43.156739    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cnw2" podUID="cdf7849a-206d-4b5a-ad81-e723c58f4426"
Apr 22 19:57:43.156850 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:43.156829    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95q85" podUID="635d0f48-b24f-422c-b672-908590754a91"
Apr 22 19:57:43.156937 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:43.156899    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:43.157033 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:43.157015    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zh2" podUID="bcfcba31-705d-4865-bba0-528576e44d02"
Apr 22 19:57:45.156477 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:45.156448    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:45.156905 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:45.156521    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:45.156905 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:45.156596    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95q85" podUID="635d0f48-b24f-422c-b672-908590754a91"
Apr 22 19:57:45.157065 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:45.156986    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zh2" podUID="bcfcba31-705d-4865-bba0-528576e44d02"
Apr 22 19:57:45.157065 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:45.157032    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:45.157156 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:45.157113    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cnw2" podUID="cdf7849a-206d-4b5a-ad81-e723c58f4426"
Apr 22 19:57:46.748758 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:46.748722    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs\") pod \"network-metrics-daemon-9cnw2\" (UID: \"cdf7849a-206d-4b5a-ad81-e723c58f4426\") " pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:46.749233 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:46.748869    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:46.749233 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:46.748929    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs podName:cdf7849a-206d-4b5a-ad81-e723c58f4426 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:02.748913952 +0000 UTC m=+34.166577580 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs") pod "network-metrics-daemon-9cnw2" (UID: "cdf7849a-206d-4b5a-ad81-e723c58f4426") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 22 19:57:46.950190 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:46.950154    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-flqgg\" (UniqueName: \"kubernetes.io/projected/635d0f48-b24f-422c-b672-908590754a91-kube-api-access-flqgg\") pod \"network-check-target-95q85\" (UID: \"635d0f48-b24f-422c-b672-908590754a91\") " pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:46.950346 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:46.950333    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 22 19:57:46.950405 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:46.950356    2574 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 22 19:57:46.950405 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:46.950369    2574 projected.go:194] Error preparing data for projected volume kube-api-access-flqgg for pod openshift-network-diagnostics/network-check-target-95q85: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:46.950538 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:46.950451    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/635d0f48-b24f-422c-b672-908590754a91-kube-api-access-flqgg podName:635d0f48-b24f-422c-b672-908590754a91 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:02.950430837 +0000 UTC m=+34.368094474 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-flqgg" (UniqueName: "kubernetes.io/projected/635d0f48-b24f-422c-b672-908590754a91-kube-api-access-flqgg") pod "network-check-target-95q85" (UID: "635d0f48-b24f-422c-b672-908590754a91") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 22 19:57:47.156710 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:47.156633    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:47.156867 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:47.156633    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:47.156867 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:47.156760    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cnw2" podUID="cdf7849a-206d-4b5a-ad81-e723c58f4426"
Apr 22 19:57:47.156867 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:47.156643    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:47.156867 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:47.156837    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zh2" podUID="bcfcba31-705d-4865-bba0-528576e44d02"
Apr 22 19:57:47.157034 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:47.156909    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95q85" podUID="635d0f48-b24f-422c-b672-908590754a91"
Apr 22 19:57:48.917564 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:57:48.917534    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4edbe2f_e978_4101_8d32_e35e7e5e740e.slice/crio-146887e8f6d11af23d3f16dfa256ffbd703344cb5d7471568398cb0f48b197d4 WatchSource:0}: Error finding container 146887e8f6d11af23d3f16dfa256ffbd703344cb5d7471568398cb0f48b197d4: Status 404 returned error can't find the container with id 146887e8f6d11af23d3f16dfa256ffbd703344cb5d7471568398cb0f48b197d4
Apr 22 19:57:48.966453 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:48.966406    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcfcba31-705d-4865-bba0-528576e44d02-original-pull-secret\") pod \"global-pull-secret-syncer-89zh2\" (UID: \"bcfcba31-705d-4865-bba0-528576e44d02\") " pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:48.966559 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:48.966509    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Apr 22 19:57:48.966613 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:48.966565    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcfcba31-705d-4865-bba0-528576e44d02-original-pull-secret podName:bcfcba31-705d-4865-bba0-528576e44d02 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:04.966551117 +0000 UTC m=+36.384214744 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bcfcba31-705d-4865-bba0-528576e44d02-original-pull-secret") pod "global-pull-secret-syncer-89zh2" (UID: "bcfcba31-705d-4865-bba0-528576e44d02") : object "kube-system"/"original-pull-secret" not registered
Apr 22 19:57:49.157806 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:49.157580    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:49.157935 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:49.157889    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95q85" podUID="635d0f48-b24f-422c-b672-908590754a91"
Apr 22 19:57:49.157993 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:49.157967    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:49.158110 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:49.158087    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cnw2" podUID="cdf7849a-206d-4b5a-ad81-e723c58f4426"
Apr 22 19:57:49.158175 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:49.158136    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:49.158229 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:49.158200    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zh2" podUID="bcfcba31-705d-4865-bba0-528576e44d02"
Apr 22 19:57:49.232341 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:49.232310    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vxzgj" event={"ID":"a56d6e25-767b-43f8-b8e0-e1dd0b4879fa","Type":"ContainerStarted","Data":"b67385e555acf1123b0c7b29d08b73785666f6b1034dc8c29f69e5b9f3a948b6"}
Apr 22 19:57:49.233635 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:49.233611    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wmn8m" event={"ID":"844c05e7-3e07-4c4a-98d5-359d7da599f2","Type":"ContainerStarted","Data":"83eee3ee97538a083afbd512c49c1d8139acc1043f18dd9f4c1b1cd8dd734d67"}
Apr 22 19:57:49.236059 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:49.235734    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js9v5" event={"ID":"7f755d56-6d83-48c7-be8f-4efe2af27fbc","Type":"ContainerStarted","Data":"08d50061c1cb6f2a93399ced8b6e3f9101b9ef48c4cfaaf5d2f636c68cac1ec0"}
Apr 22 19:57:49.237164 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:49.236975    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wgvqq" event={"ID":"5063845e-1c48-48a7-bfa3-b31c3d6f8109","Type":"ContainerStarted","Data":"2779d8e9719a09ff7edeaccde882da2970b6ec2d153cdc01d8d28e258f501c5d"}
Apr 22 19:57:49.238541 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:49.238509    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7ttkd" event={"ID":"84c4bd83-8750-4357-8974-fa2ff745d809","Type":"ContainerStarted","Data":"e04c29604c42af292e49919fdf935cc974f00638db12cee85e5db9131fa19179"}
Apr 22 19:57:49.239678 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:49.239655    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sz29m" event={"ID":"c4edbe2f-e978-4101-8d32-e35e7e5e740e","Type":"ContainerStarted","Data":"73f9c055160bd09a12f6fe456f04ae7ae82edf2c1ac7a0d3f9d681cf63a54638"}
Apr 22 19:57:49.239759 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:49.239684    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sz29m" event={"ID":"c4edbe2f-e978-4101-8d32-e35e7e5e740e","Type":"ContainerStarted","Data":"146887e8f6d11af23d3f16dfa256ffbd703344cb5d7471568398cb0f48b197d4"}
Apr 22 19:57:49.254222 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:49.254183    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wmn8m" podStartSLOduration=3.312380769 podStartE2EDuration="20.254169723s" podCreationTimestamp="2026-04-22 19:57:29 +0000 UTC" firstStartedPulling="2026-04-22 19:57:31.92470139 +0000 UTC m=+3.342365018" lastFinishedPulling="2026-04-22 19:57:48.866490333 +0000 UTC m=+20.284153972" observedRunningTime="2026-04-22 19:57:49.254062457 +0000 UTC m=+20.671726103" watchObservedRunningTime="2026-04-22 19:57:49.254169723 +0000 UTC m=+20.671833368"
Apr 22 19:57:49.254409 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:49.254377    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vxzgj" podStartSLOduration=3.313708364 podStartE2EDuration="20.254366979s" podCreationTimestamp="2026-04-22 19:57:29 +0000 UTC" firstStartedPulling="2026-04-22 19:57:31.925855105 +0000 UTC m=+3.343518731" lastFinishedPulling="2026-04-22 19:57:48.866513706 +0000 UTC m=+20.284177346" observedRunningTime="2026-04-22 19:57:49.243682765 +0000 UTC m=+20.661346412" watchObservedRunningTime="2026-04-22 19:57:49.254366979 +0000 UTC m=+20.672030625"
Apr 22 19:57:49.268543 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:49.268512    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wgvqq" podStartSLOduration=3.319477495 podStartE2EDuration="20.26850083s" podCreationTimestamp="2026-04-22 19:57:29 +0000 UTC" firstStartedPulling="2026-04-22 19:57:31.917605839 +0000 UTC m=+3.335269463" lastFinishedPulling="2026-04-22 19:57:48.866629175 +0000 UTC m=+20.284292798" observedRunningTime="2026-04-22 19:57:49.26803269 +0000 UTC m=+20.685696335" watchObservedRunningTime="2026-04-22 19:57:49.26850083 +0000 UTC m=+20.686164475"
Apr 22 19:57:49.302076 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:49.302029    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-7ttkd" podStartSLOduration=3.293508144 podStartE2EDuration="20.302017836s" podCreationTimestamp="2026-04-22 19:57:29 +0000 UTC" firstStartedPulling="2026-04-22 19:57:31.916729802 +0000 UTC m=+3.334393429" lastFinishedPulling="2026-04-22 19:57:48.925239497 +0000 UTC m=+20.342903121" observedRunningTime="2026-04-22 19:57:49.287119689 +0000 UTC m=+20.704783336" watchObservedRunningTime="2026-04-22 19:57:49.302017836 +0000 UTC m=+20.719681482"
Apr 22 19:57:50.242927 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:50.242734    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p" event={"ID":"c9e91d24-9315-418d-98d2-9750bf166aa2","Type":"ContainerStarted","Data":"d14e16add26d0751bf40a724962fa67a8df20c124d6cca05fdc9524d2a3c2a55"}
Apr 22 19:57:50.243651 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:50.242943    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p" event={"ID":"c9e91d24-9315-418d-98d2-9750bf166aa2","Type":"ContainerStarted","Data":"f6137da1e9a6d99f53cd4f34760a39af7ef23bffc6727d2585b686184df0c8fe"}
Apr 22 19:57:50.245815 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:50.245791    2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Apr 22 19:57:50.245981 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:50.245962    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4" event={"ID":"d54b3b0c-260b-4004-95d8-2d981031a8a0","Type":"ContainerStarted","Data":"ea0f83f3bb39367b6c92c7b0880205100ca57f26c4dd77cc89384d4ca57943c3"}
Apr 22 19:57:50.246053 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:50.245990    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4" event={"ID":"d54b3b0c-260b-4004-95d8-2d981031a8a0","Type":"ContainerStarted","Data":"65294b7a8c126795bddedd4dca8a4eb08e3a79ad65ab715e11421546a8c607bf"}
Apr 22 19:57:50.246053 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:50.246000    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4" event={"ID":"d54b3b0c-260b-4004-95d8-2d981031a8a0","Type":"ContainerStarted","Data":"ed8b0e9e824ba2beeed1087098c3cac97e4384bea28083f3306fb37ee60b051f"}
Apr 22 19:57:50.246053 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:50.246008    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4" event={"ID":"d54b3b0c-260b-4004-95d8-2d981031a8a0","Type":"ContainerStarted","Data":"128cce48271bc0c05b84de48d99358449cdc89698de6d3a31f8d19b859678fb9"}
Apr 22 19:57:50.246053 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:50.246018    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4" event={"ID":"d54b3b0c-260b-4004-95d8-2d981031a8a0","Type":"ContainerStarted","Data":"3c5da78536907b34e49374319142f25796c4d959c1a01c6aefadc61286fa9f69"}
Apr 22 19:57:50.246053 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:50.246028    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4" event={"ID":"d54b3b0c-260b-4004-95d8-2d981031a8a0","Type":"ContainerStarted","Data":"ff95c5831606c842ee9ddd2505678f127247075a9053c80820672a4540d083e8"}
Apr 22 19:57:50.247566 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:50.247539    2574 generic.go:358] "Generic (PLEG): container finished" podID="7f755d56-6d83-48c7-be8f-4efe2af27fbc" containerID="08d50061c1cb6f2a93399ced8b6e3f9101b9ef48c4cfaaf5d2f636c68cac1ec0" exitCode=0
Apr 22 19:57:50.247659 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:50.247577    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js9v5" event={"ID":"7f755d56-6d83-48c7-be8f-4efe2af27fbc","Type":"ContainerDied","Data":"08d50061c1cb6f2a93399ced8b6e3f9101b9ef48c4cfaaf5d2f636c68cac1ec0"}
Apr 22 19:57:50.621888 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:50.621704    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wmn8m"
Apr 22 19:57:50.622299 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:50.622282    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wmn8m"
Apr 22 19:57:50.636181 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:50.636135    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-sz29m" podStartSLOduration=8.636118468 podStartE2EDuration="8.636118468s" podCreationTimestamp="2026-04-22 19:57:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 19:57:50.27287258 +0000 UTC m=+21.690536225" watchObservedRunningTime="2026-04-22 19:57:50.636118468 +0000 UTC m=+22.053782115"
Apr 22 19:57:51.074685 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:51.074613    2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-04-22T19:57:50.245810774Z","UUID":"c91d3976-3550-425c-a759-609801d78df5","Handler":null,"Name":"","Endpoint":""}
Apr 22 19:57:51.077498 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:51.077476    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Apr 22 19:57:51.077617 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:51.077508    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Apr 22 19:57:51.156536 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:51.156511    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:51.156676 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:51.156511    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:51.156676 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:51.156615    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zh2" podUID="bcfcba31-705d-4865-bba0-528576e44d02"
Apr 22 19:57:51.156789 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:51.156717    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95q85" podUID="635d0f48-b24f-422c-b672-908590754a91"
Apr 22 19:57:51.156789 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:51.156511    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:51.156899 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:51.156820    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cnw2" podUID="cdf7849a-206d-4b5a-ad81-e723c58f4426"
Apr 22 19:57:51.251165 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:51.251093    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p" event={"ID":"c9e91d24-9315-418d-98d2-9750bf166aa2","Type":"ContainerStarted","Data":"c88c69d352812a68639cf83393870cf3ddc586d14c85a1320f9c020b048bda49"}
Apr 22 19:57:51.252392 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:51.252367    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8ht5v" event={"ID":"2dca392c-613c-49fb-a976-970fe078c0c4","Type":"ContainerStarted","Data":"81155e46743bdb706c23052b2130184af74a49c5fc5a115fc81f5a54d4423dde"}
Apr 22 19:57:51.252635 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:51.252619    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wmn8m"
Apr 22 19:57:51.253031 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:51.253017    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wmn8m"
Apr 22 19:57:51.275822 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:51.275779    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr45p" podStartSLOduration=3.240871871 podStartE2EDuration="22.275764199s" podCreationTimestamp="2026-04-22 19:57:29 +0000 UTC" firstStartedPulling="2026-04-22 19:57:31.921961706 +0000 UTC m=+3.339625338" lastFinishedPulling="2026-04-22 19:57:50.956854025 +0000 UTC m=+22.374517666" observedRunningTime="2026-04-22 19:57:51.275563217 +0000 UTC m=+22.693226862" watchObservedRunningTime="2026-04-22 19:57:51.275764199 +0000 UTC m=+22.693427847"
Apr 22 19:57:51.288809 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:51.288770    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8ht5v" podStartSLOduration=5.33834096 podStartE2EDuration="22.288759154s" podCreationTimestamp="2026-04-22 19:57:29 +0000 UTC" firstStartedPulling="2026-04-22 19:57:31.915964359 +0000 UTC m=+3.333627998" lastFinishedPulling="2026-04-22 19:57:48.866382557 +0000 UTC m=+20.284046192" observedRunningTime="2026-04-22 19:57:51.288151611 +0000 UTC m=+22.705815256" watchObservedRunningTime="2026-04-22 19:57:51.288759154 +0000 UTC m=+22.706422796"
Apr 22 19:57:52.257302 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:52.257257    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4" event={"ID":"d54b3b0c-260b-4004-95d8-2d981031a8a0","Type":"ContainerStarted","Data":"c512e04fb295a0d6da390638556b440212dd1d11098c5cdc776db42da618791a"}
Apr 22 19:57:53.155934 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:53.155901    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:53.156120 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:53.155982    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:53.156120 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:53.156002    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:53.156120 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:53.156101    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cnw2" podUID="cdf7849a-206d-4b5a-ad81-e723c58f4426"
Apr 22 19:57:53.156397 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:53.156362    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zh2" podUID="bcfcba31-705d-4865-bba0-528576e44d02"
Apr 22 19:57:53.156506 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:53.156470    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95q85" podUID="635d0f48-b24f-422c-b672-908590754a91"
Apr 22 19:57:55.156054 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:55.155920    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:55.156574 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:55.156123    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95q85" podUID="635d0f48-b24f-422c-b672-908590754a91"
Apr 22 19:57:55.156574 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:55.155932    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:55.156574 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:55.156269    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cnw2" podUID="cdf7849a-206d-4b5a-ad81-e723c58f4426"
Apr 22 19:57:55.156574 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:55.155932    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:55.156574 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:55.156371    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zh2" podUID="bcfcba31-705d-4865-bba0-528576e44d02"
Apr 22 19:57:55.263629 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:55.263599    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js9v5" event={"ID":"7f755d56-6d83-48c7-be8f-4efe2af27fbc","Type":"ContainerStarted","Data":"d9dda19633b6278d23e5aef38ef309a5d1dc26b9f3c611b4741732bba592b677"}
Apr 22 19:57:55.268115 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:55.268088    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4" event={"ID":"d54b3b0c-260b-4004-95d8-2d981031a8a0","Type":"ContainerStarted","Data":"4609d55bcbac361fcf5e5d0631c44b409a3646618b82d6814bf7636fcfcb29bb"}
Apr 22 19:57:55.268443 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:55.268401    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:55.268533 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:55.268461    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:55.268533 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:55.268472    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:55.291493 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:55.291460    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:55.291666 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:55.291649    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:57:55.313174 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:55.313133    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4" podStartSLOduration=8.85772541 podStartE2EDuration="26.3131217s" podCreationTimestamp="2026-04-22 19:57:29 +0000 UTC" firstStartedPulling="2026-04-22 19:57:31.920851684 +0000 UTC m=+3.338515313" lastFinishedPulling="2026-04-22 19:57:49.376247966 +0000 UTC m=+20.793911603" observedRunningTime="2026-04-22 19:57:55.312733354 +0000 UTC m=+26.730397000" watchObservedRunningTime="2026-04-22 19:57:55.3131217 +0000 UTC m=+26.730785345"
Apr 22 19:57:56.271317 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:56.271281    2574 generic.go:358] "Generic (PLEG): container finished" podID="7f755d56-6d83-48c7-be8f-4efe2af27fbc" containerID="d9dda19633b6278d23e5aef38ef309a5d1dc26b9f3c611b4741732bba592b677" exitCode=0
Apr 22 19:57:56.271683 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:56.271370    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js9v5" event={"ID":"7f755d56-6d83-48c7-be8f-4efe2af27fbc","Type":"ContainerDied","Data":"d9dda19633b6278d23e5aef38ef309a5d1dc26b9f3c611b4741732bba592b677"}
Apr 22 19:57:56.627158 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:56.627073    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-89zh2"]
Apr 22 19:57:56.627288 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:56.627213    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:56.627328 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:56.627310    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zh2" podUID="bcfcba31-705d-4865-bba0-528576e44d02"
Apr 22 19:57:56.630264 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:56.630244    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9cnw2"]
Apr 22 19:57:56.630360 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:56.630340    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:56.630448 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:56.630432    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cnw2" podUID="cdf7849a-206d-4b5a-ad81-e723c58f4426"
Apr 22 19:57:56.630703 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:56.630687    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-95q85"]
Apr 22 19:57:56.630801 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:56.630788    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:56.630882 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:56.630864    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95q85" podUID="635d0f48-b24f-422c-b672-908590754a91"
Apr 22 19:57:58.156358 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:58.156196    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:57:58.156703 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:58.156268    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:57:58.156703 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:58.156453    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95q85" podUID="635d0f48-b24f-422c-b672-908590754a91"
Apr 22 19:57:58.156703 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:58.156298    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:57:58.156703 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:58.156528    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zh2" podUID="bcfcba31-705d-4865-bba0-528576e44d02"
Apr 22 19:57:58.156703 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:57:58.156597    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cnw2" podUID="cdf7849a-206d-4b5a-ad81-e723c58f4426"
Apr 22 19:57:58.277907 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:58.277875    2574 generic.go:358] "Generic (PLEG): container finished" podID="7f755d56-6d83-48c7-be8f-4efe2af27fbc" containerID="24c9a648f72481b1c643d4dafc5608a9c7da521327fee5514875717037e664db" exitCode=0
Apr 22 19:57:58.278046 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:58.277927    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js9v5" event={"ID":"7f755d56-6d83-48c7-be8f-4efe2af27fbc","Type":"ContainerDied","Data":"24c9a648f72481b1c643d4dafc5608a9c7da521327fee5514875717037e664db"}
Apr 22 19:57:59.281200 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:59.281172    2574 generic.go:358] "Generic (PLEG): container finished" podID="7f755d56-6d83-48c7-be8f-4efe2af27fbc" containerID="d18ab19c3dcd5480e45dcd30f166abb8207c81a860a653d1223046246f60e55f" exitCode=0
Apr 22 19:57:59.281657 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:57:59.281219    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js9v5" event={"ID":"7f755d56-6d83-48c7-be8f-4efe2af27fbc","Type":"ContainerDied","Data":"d18ab19c3dcd5480e45dcd30f166abb8207c81a860a653d1223046246f60e55f"}
Apr 22 19:58:00.155734 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:00.155702    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:58:00.155897 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:00.155708    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:58:00.155897 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:00.155820    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zh2" podUID="bcfcba31-705d-4865-bba0-528576e44d02"
Apr 22 19:58:00.155897 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:00.155714    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:58:00.156043 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:00.155904    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95q85" podUID="635d0f48-b24f-422c-b672-908590754a91"
Apr 22 19:58:00.156043 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:00.155974    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cnw2" podUID="cdf7849a-206d-4b5a-ad81-e723c58f4426"
Apr 22 19:58:01.946489 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:01.946462    2574 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-215.ec2.internal" event="NodeReady"
Apr 22 19:58:01.947254 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:01.946610    2574 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
Apr 22 19:58:01.978761 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:01.978719    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"]
Apr 22 19:58:01.981739 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:01.981711    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"
Apr 22 19:58:01.983465 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:01.983441    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59cb695579-cvq2f"]
Apr 22 19:58:01.985375 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:01.984669    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-864n4\""
Apr 22 19:58:01.985375 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:01.984902    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Apr 22 19:58:01.985375 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:01.985044    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Apr 22 19:58:01.986274 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:01.986254    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:01.988701 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:01.988636    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fs69p"]
Apr 22 19:58:01.988940 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:01.988917    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Apr 22 19:58:01.989097 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:01.989084    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Apr 22 19:58:01.989344 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:01.989330    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Apr 22 19:58:01.989478 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:01.989461    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-glwlc\""
Apr 22 19:58:01.992669 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:01.992076    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fs69p"
Apr 22 19:58:01.996935 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:01.996911    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Apr 22 19:58:01.998330 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:01.997877    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Apr 22 19:58:01.998330 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:01.997927    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tzbrr\""
Apr 22 19:58:01.999337 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:01.999256    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"]
Apr 22 19:58:01.999723 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:01.999701    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Apr 22 19:58:01.999822 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:01.999810    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ct7n5"]
Apr 22 19:58:02.003541 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.002787    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59cb695579-cvq2f"]
Apr 22 19:58:02.003541 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.002886    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ct7n5"
Apr 22 19:58:02.003541 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.003495    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ct7n5"]
Apr 22 19:58:02.006005 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.005661    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Apr 22 19:58:02.006005 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.005724    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Apr 22 19:58:02.006170 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.006023    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Apr 22 19:58:02.006614 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.006593    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lxbw7\""
Apr 22 19:58:02.009323 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.009301    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fs69p"]
Apr 22 19:58:02.155838 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.155793    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:58:02.155838 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.155829    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:58:02.156086 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.156005    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:58:02.159004 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.158838    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Apr 22 19:58:02.159004 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.158847    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-79t2x\""
Apr 22 19:58:02.159004 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.158858    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cbww9\""
Apr 22 19:58:02.159004 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.158852    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Apr 22 19:58:02.159004 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.158976    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Apr 22 19:58:02.159315 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.159182    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Apr 22 19:58:02.169475 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.169449    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-27n8r\" (UID: \"173cdd12-cd9e-403c-a2b2-e18a20be58a4\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"
Apr 22 19:58:02.169600 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.169495    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/173cdd12-cd9e-403c-a2b2-e18a20be58a4-nginx-conf\") pod \"networking-console-plugin-cb95c66f6-27n8r\" (UID: \"173cdd12-cd9e-403c-a2b2-e18a20be58a4\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"
Apr 22 19:58:02.169600 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.169527    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-bound-sa-token\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.169600 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.169552    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/054ccdff-220c-47af-b943-4797d37b8723-config-volume\") pod \"dns-default-fs69p\" (UID: \"054ccdff-220c-47af-b943-4797d37b8723\") " pod="openshift-dns/dns-default-fs69p"
Apr 22 19:58:02.169600 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.169577    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/054ccdff-220c-47af-b943-4797d37b8723-tmp-dir\") pod \"dns-default-fs69p\" (UID: \"054ccdff-220c-47af-b943-4797d37b8723\") " pod="openshift-dns/dns-default-fs69p"
Apr 22 19:58:02.169600 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.169597    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert\") pod \"ingress-canary-ct7n5\" (UID: \"3a7698c5-bde7-4c7c-84ac-bcdba6d86851\") " pod="openshift-ingress-canary/ingress-canary-ct7n5"
Apr 22 19:58:02.169755 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.169627    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/819fb49a-5bbf-4c97-ad55-ca03015622ca-ca-trust-extracted\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.169755 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.169652    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/819fb49a-5bbf-4c97-ad55-ca03015622ca-installation-pull-secrets\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.169755 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.169680    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7r4r\" (UniqueName: \"kubernetes.io/projected/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-kube-api-access-k7r4r\") pod \"ingress-canary-ct7n5\" (UID: \"3a7698c5-bde7-4c7c-84ac-bcdba6d86851\") " pod="openshift-ingress-canary/ingress-canary-ct7n5"
Apr 22 19:58:02.169755 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.169718    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/819fb49a-5bbf-4c97-ad55-ca03015622ca-image-registry-private-configuration\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.169937 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.169759    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.169937 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.169782    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-certificates\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.169937 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.169846    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffwct\" (UniqueName: \"kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-kube-api-access-ffwct\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.169937 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.169905    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls\") pod \"dns-default-fs69p\" (UID: \"054ccdff-220c-47af-b943-4797d37b8723\") " pod="openshift-dns/dns-default-fs69p"
Apr 22 19:58:02.170126 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.169955    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/819fb49a-5bbf-4c97-ad55-ca03015622ca-trusted-ca\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.170126 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.169987    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgzdm\" (UniqueName: \"kubernetes.io/projected/054ccdff-220c-47af-b943-4797d37b8723-kube-api-access-mgzdm\") pod \"dns-default-fs69p\" (UID: \"054ccdff-220c-47af-b943-4797d37b8723\") " pod="openshift-dns/dns-default-fs69p"
Apr 22 19:58:02.271214 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.271141    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.271214 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.271182    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-certificates\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.271489 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.271250    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ffwct\" (UniqueName: \"kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-kube-api-access-ffwct\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.271489 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.271279    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls\") pod \"dns-default-fs69p\" (UID: \"054ccdff-220c-47af-b943-4797d37b8723\") " pod="openshift-dns/dns-default-fs69p"
Apr 22 19:58:02.271489 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:02.271291    2574 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Apr 22 19:58:02.271489 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:02.271310    2574 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59cb695579-cvq2f: secret "image-registry-tls" not found
Apr 22 19:58:02.271489 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:02.271383    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls podName:819fb49a-5bbf-4c97-ad55-ca03015622ca nodeName:}" failed. No retries permitted until 2026-04-22 19:58:02.771361812 +0000 UTC m=+34.189025440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls") pod "image-registry-59cb695579-cvq2f" (UID: "819fb49a-5bbf-4c97-ad55-ca03015622ca") : secret "image-registry-tls" not found
Apr 22 19:58:02.271489 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.271307    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/819fb49a-5bbf-4c97-ad55-ca03015622ca-trusted-ca\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.271489 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:02.271465    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 22 19:58:02.271489 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.271472    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mgzdm\" (UniqueName: \"kubernetes.io/projected/054ccdff-220c-47af-b943-4797d37b8723-kube-api-access-mgzdm\") pod \"dns-default-fs69p\" (UID: \"054ccdff-220c-47af-b943-4797d37b8723\") " pod="openshift-dns/dns-default-fs69p"
Apr 22 19:58:02.271888 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:02.271538    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls podName:054ccdff-220c-47af-b943-4797d37b8723 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:02.771517773 +0000 UTC m=+34.189181401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls") pod "dns-default-fs69p" (UID: "054ccdff-220c-47af-b943-4797d37b8723") : secret "dns-default-metrics-tls" not found
Apr 22 19:58:02.271888 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.271585    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-27n8r\" (UID: \"173cdd12-cd9e-403c-a2b2-e18a20be58a4\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"
Apr 22 19:58:02.271888 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.271624    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/173cdd12-cd9e-403c-a2b2-e18a20be58a4-nginx-conf\") pod \"networking-console-plugin-cb95c66f6-27n8r\" (UID: \"173cdd12-cd9e-403c-a2b2-e18a20be58a4\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"
Apr 22 19:58:02.271888 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.271654    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-bound-sa-token\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.271888 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.271682    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/054ccdff-220c-47af-b943-4797d37b8723-config-volume\") pod \"dns-default-fs69p\" (UID: \"054ccdff-220c-47af-b943-4797d37b8723\") " pod="openshift-dns/dns-default-fs69p"
Apr 22 19:58:02.271888 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.271709    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/054ccdff-220c-47af-b943-4797d37b8723-tmp-dir\") pod \"dns-default-fs69p\" (UID: \"054ccdff-220c-47af-b943-4797d37b8723\") " pod="openshift-dns/dns-default-fs69p"
Apr 22 19:58:02.271888 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:02.271731    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Apr 22 19:58:02.271888 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.271736    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert\") pod \"ingress-canary-ct7n5\" (UID: \"3a7698c5-bde7-4c7c-84ac-bcdba6d86851\") " pod="openshift-ingress-canary/ingress-canary-ct7n5"
Apr 22 19:58:02.271888 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.271778    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/819fb49a-5bbf-4c97-ad55-ca03015622ca-ca-trust-extracted\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.271888 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:02.271792    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert podName:173cdd12-cd9e-403c-a2b2-e18a20be58a4 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:02.771775703 +0000 UTC m=+34.189439327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-27n8r" (UID: "173cdd12-cd9e-403c-a2b2-e18a20be58a4") : secret "networking-console-plugin-cert" not found
Apr 22 19:58:02.271888 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.271828    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/819fb49a-5bbf-4c97-ad55-ca03015622ca-installation-pull-secrets\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.271888 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.271869    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k7r4r\" (UniqueName: \"kubernetes.io/projected/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-kube-api-access-k7r4r\") pod \"ingress-canary-ct7n5\" (UID: \"3a7698c5-bde7-4c7c-84ac-bcdba6d86851\") " pod="openshift-ingress-canary/ingress-canary-ct7n5"
Apr 22 19:58:02.272547 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.271909    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/819fb49a-5bbf-4c97-ad55-ca03015622ca-image-registry-private-configuration\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.272547 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.272129    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/819fb49a-5bbf-4c97-ad55-ca03015622ca-ca-trust-extracted\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.272547 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.272310    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-certificates\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.272547 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.272348    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/819fb49a-5bbf-4c97-ad55-ca03015622ca-trusted-ca\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.272547 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:02.272450    2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 22 19:58:02.272547 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:02.272503    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert podName:3a7698c5-bde7-4c7c-84ac-bcdba6d86851 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:02.772486459 +0000 UTC m=+34.190150096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert") pod "ingress-canary-ct7n5" (UID: "3a7698c5-bde7-4c7c-84ac-bcdba6d86851") : secret "canary-serving-cert" not found
Apr 22 19:58:02.272887 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.272705    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/054ccdff-220c-47af-b943-4797d37b8723-tmp-dir\") pod \"dns-default-fs69p\" (UID: \"054ccdff-220c-47af-b943-4797d37b8723\") " pod="openshift-dns/dns-default-fs69p"
Apr 22 19:58:02.272887 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.272880    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/054ccdff-220c-47af-b943-4797d37b8723-config-volume\") pod \"dns-default-fs69p\" (UID: \"054ccdff-220c-47af-b943-4797d37b8723\") " pod="openshift-dns/dns-default-fs69p"
Apr 22 19:58:02.273093 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.273073    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/173cdd12-cd9e-403c-a2b2-e18a20be58a4-nginx-conf\") pod \"networking-console-plugin-cb95c66f6-27n8r\" (UID: \"173cdd12-cd9e-403c-a2b2-e18a20be58a4\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"
Apr 22 19:58:02.276297 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.276276    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/819fb49a-5bbf-4c97-ad55-ca03015622ca-installation-pull-secrets\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.276432 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.276394    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/819fb49a-5bbf-4c97-ad55-ca03015622ca-image-registry-private-configuration\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.282886 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.282633    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgzdm\" (UniqueName: \"kubernetes.io/projected/054ccdff-220c-47af-b943-4797d37b8723-kube-api-access-mgzdm\") pod \"dns-default-fs69p\" (UID: \"054ccdff-220c-47af-b943-4797d37b8723\") " pod="openshift-dns/dns-default-fs69p"
Apr 22 19:58:02.283337 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.283313    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-bound-sa-token\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.283969 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.283947    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7r4r\" (UniqueName: \"kubernetes.io/projected/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-kube-api-access-k7r4r\") pod \"ingress-canary-ct7n5\" (UID: \"3a7698c5-bde7-4c7c-84ac-bcdba6d86851\") " pod="openshift-ingress-canary/ingress-canary-ct7n5"
Apr 22 19:58:02.284986 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.284964    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffwct\" (UniqueName: \"kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-kube-api-access-ffwct\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.775387 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.775357    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert\") pod \"ingress-canary-ct7n5\" (UID: \"3a7698c5-bde7-4c7c-84ac-bcdba6d86851\") " pod="openshift-ingress-canary/ingress-canary-ct7n5"
Apr 22 19:58:02.775544 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.775423    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs\") pod \"network-metrics-daemon-9cnw2\" (UID: \"cdf7849a-206d-4b5a-ad81-e723c58f4426\") " pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:58:02.775544 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.775445    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:02.775544 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.775475    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls\") pod \"dns-default-fs69p\" (UID: \"054ccdff-220c-47af-b943-4797d37b8723\") " pod="openshift-dns/dns-default-fs69p"
Apr 22 19:58:02.775544 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.775512    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-27n8r\" (UID: \"173cdd12-cd9e-403c-a2b2-e18a20be58a4\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"
Apr 22 19:58:02.775544 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:02.775510    2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 22 19:58:02.775746 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:02.775555    2574 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Apr 22 19:58:02.775746 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:02.775563    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 22 19:58:02.775746 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:02.775569    2574 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59cb695579-cvq2f: secret "image-registry-tls" not found
Apr 22 19:58:02.775746 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:02.775521    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Apr 22 19:58:02.775746 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:02.775608    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert podName:3a7698c5-bde7-4c7c-84ac-bcdba6d86851 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:03.775589744 +0000 UTC m=+35.193253368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert") pod "ingress-canary-ct7n5" (UID: "3a7698c5-bde7-4c7c-84ac-bcdba6d86851") : secret "canary-serving-cert" not found
Apr 22 19:58:02.775746 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:02.775603    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Apr 22 19:58:02.775746 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:02.775626    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls podName:054ccdff-220c-47af-b943-4797d37b8723 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:03.775616363 +0000 UTC m=+35.193279986 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls") pod "dns-default-fs69p" (UID: "054ccdff-220c-47af-b943-4797d37b8723") : secret "dns-default-metrics-tls" not found
Apr 22 19:58:02.776050 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:02.776017    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls podName:819fb49a-5bbf-4c97-ad55-ca03015622ca nodeName:}" failed. No retries permitted until 2026-04-22 19:58:03.775892357 +0000 UTC m=+35.193555981 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls") pod "image-registry-59cb695579-cvq2f" (UID: "819fb49a-5bbf-4c97-ad55-ca03015622ca") : secret "image-registry-tls" not found
Apr 22 19:58:02.776101 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:02.776063    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs podName:cdf7849a-206d-4b5a-ad81-e723c58f4426 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:34.776029363 +0000 UTC m=+66.193692989 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs") pod "network-metrics-daemon-9cnw2" (UID: "cdf7849a-206d-4b5a-ad81-e723c58f4426") : secret "metrics-daemon-secret" not found
Apr 22 19:58:02.776101 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:02.776090    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert podName:173cdd12-cd9e-403c-a2b2-e18a20be58a4 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:03.776071494 +0000 UTC m=+35.193735117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-27n8r" (UID: "173cdd12-cd9e-403c-a2b2-e18a20be58a4") : secret "networking-console-plugin-cert" not found
Apr 22 19:58:02.977135 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.977099    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-flqgg\" (UniqueName: \"kubernetes.io/projected/635d0f48-b24f-422c-b672-908590754a91-kube-api-access-flqgg\") pod \"network-check-target-95q85\" (UID: \"635d0f48-b24f-422c-b672-908590754a91\") " pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:58:02.980606 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:02.980578    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flqgg\" (UniqueName: \"kubernetes.io/projected/635d0f48-b24f-422c-b672-908590754a91-kube-api-access-flqgg\") pod \"network-check-target-95q85\" (UID: \"635d0f48-b24f-422c-b672-908590754a91\") " pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:58:03.082220 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:03.082004    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:58:03.784358 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:03.784320    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-27n8r\" (UID: \"173cdd12-cd9e-403c-a2b2-e18a20be58a4\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"
Apr 22 19:58:03.784358 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:03.784369    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert\") pod \"ingress-canary-ct7n5\" (UID: \"3a7698c5-bde7-4c7c-84ac-bcdba6d86851\") " pod="openshift-ingress-canary/ingress-canary-ct7n5"
Apr 22 19:58:03.784642 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:03.784409    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:03.784642 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:03.784452    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls\") pod \"dns-default-fs69p\" (UID: \"054ccdff-220c-47af-b943-4797d37b8723\") " pod="openshift-dns/dns-default-fs69p"
Apr 22 19:58:03.784642 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:03.784507    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Apr 22 19:58:03.784642 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:03.784544    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 22 19:58:03.784642 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:03.784546    2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 22 19:58:03.784642 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:03.784549    2574 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Apr 22 19:58:03.784642 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:03.784601    2574 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59cb695579-cvq2f: secret "image-registry-tls" not found
Apr 22 19:58:03.784642 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:03.784590    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert podName:173cdd12-cd9e-403c-a2b2-e18a20be58a4 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:05.78456967 +0000 UTC m=+37.202233303 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-27n8r" (UID: "173cdd12-cd9e-403c-a2b2-e18a20be58a4") : secret "networking-console-plugin-cert" not found
Apr 22 19:58:03.784642 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:03.784647    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls podName:054ccdff-220c-47af-b943-4797d37b8723 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:05.784629105 +0000 UTC m=+37.202292733 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls") pod "dns-default-fs69p" (UID: "054ccdff-220c-47af-b943-4797d37b8723") : secret "dns-default-metrics-tls" not found
Apr 22 19:58:03.785079 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:03.784663    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert podName:3a7698c5-bde7-4c7c-84ac-bcdba6d86851 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:05.784657237 +0000 UTC m=+37.202320860 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert") pod "ingress-canary-ct7n5" (UID: "3a7698c5-bde7-4c7c-84ac-bcdba6d86851") : secret "canary-serving-cert" not found
Apr 22 19:58:03.785079 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:03.784676    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls podName:819fb49a-5bbf-4c97-ad55-ca03015622ca nodeName:}" failed. No retries permitted until 2026-04-22 19:58:05.784671602 +0000 UTC m=+37.202335225 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls") pod "image-registry-59cb695579-cvq2f" (UID: "819fb49a-5bbf-4c97-ad55-ca03015622ca") : secret "image-registry-tls" not found
Apr 22 19:58:04.992233 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:04.992196    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcfcba31-705d-4865-bba0-528576e44d02-original-pull-secret\") pod \"global-pull-secret-syncer-89zh2\" (UID: \"bcfcba31-705d-4865-bba0-528576e44d02\") " pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:58:04.995373 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:04.995343    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcfcba31-705d-4865-bba0-528576e44d02-original-pull-secret\") pod \"global-pull-secret-syncer-89zh2\" (UID: \"bcfcba31-705d-4865-bba0-528576e44d02\") " pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:58:05.167869 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:05.167828    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zh2"
Apr 22 19:58:05.255329 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:05.255299    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-95q85"]
Apr 22 19:58:05.293705 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:05.293676    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-89zh2"]
Apr 22 19:58:05.325693 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:58:05.325668    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635d0f48_b24f_422c_b672_908590754a91.slice/crio-6642a092a049d2dfef6ebb2768d7fce15e56ce11def4dc185c8575d3c600f56e WatchSource:0}: Error finding container 6642a092a049d2dfef6ebb2768d7fce15e56ce11def4dc185c8575d3c600f56e: Status 404 returned error can't find the container with id 6642a092a049d2dfef6ebb2768d7fce15e56ce11def4dc185c8575d3c600f56e
Apr 22 19:58:05.326308 ip-10-0-135-215 kubenswrapper[2574]: W0422 19:58:05.326288    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfcba31_705d_4865_bba0_528576e44d02.slice/crio-e40671900ac2e7b372a34fb27b12e95392938a21503883f5dd0a4325d80f0f87 WatchSource:0}: Error finding container e40671900ac2e7b372a34fb27b12e95392938a21503883f5dd0a4325d80f0f87: Status 404 returned error can't find the container with id e40671900ac2e7b372a34fb27b12e95392938a21503883f5dd0a4325d80f0f87
Apr 22 19:58:05.798683 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:05.798621    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:05.798683 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:05.798654    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls\") pod \"dns-default-fs69p\" (UID: \"054ccdff-220c-47af-b943-4797d37b8723\") " pod="openshift-dns/dns-default-fs69p"
Apr 22 19:58:05.798875 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:05.798688    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-27n8r\" (UID: \"173cdd12-cd9e-403c-a2b2-e18a20be58a4\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"
Apr 22 19:58:05.798875 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:05.798724    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert\") pod \"ingress-canary-ct7n5\" (UID: \"3a7698c5-bde7-4c7c-84ac-bcdba6d86851\") " pod="openshift-ingress-canary/ingress-canary-ct7n5"
Apr 22 19:58:05.798875 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:05.798770    2574 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Apr 22 19:58:05.798875 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:05.798785    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 22 19:58:05.798875 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:05.798790    2574 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59cb695579-cvq2f: secret "image-registry-tls" not found
Apr 22 19:58:05.798875 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:05.798810    2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 22 19:58:05.798875 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:05.798847    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls podName:819fb49a-5bbf-4c97-ad55-ca03015622ca nodeName:}" failed. No retries permitted until 2026-04-22 19:58:09.79883248 +0000 UTC m=+41.216496104 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls") pod "image-registry-59cb695579-cvq2f" (UID: "819fb49a-5bbf-4c97-ad55-ca03015622ca") : secret "image-registry-tls" not found
Apr 22 19:58:05.798875 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:05.798860    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert podName:3a7698c5-bde7-4c7c-84ac-bcdba6d86851 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:09.798854636 +0000 UTC m=+41.216518260 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert") pod "ingress-canary-ct7n5" (UID: "3a7698c5-bde7-4c7c-84ac-bcdba6d86851") : secret "canary-serving-cert" not found
Apr 22 19:58:05.798875 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:05.798870    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls podName:054ccdff-220c-47af-b943-4797d37b8723 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:09.798864979 +0000 UTC m=+41.216528603 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls") pod "dns-default-fs69p" (UID: "054ccdff-220c-47af-b943-4797d37b8723") : secret "dns-default-metrics-tls" not found
Apr 22 19:58:05.798875 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:05.798877    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Apr 22 19:58:05.799204 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:05.798940    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert podName:173cdd12-cd9e-403c-a2b2-e18a20be58a4 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:09.798922048 +0000 UTC m=+41.216585673 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-27n8r" (UID: "173cdd12-cd9e-403c-a2b2-e18a20be58a4") : secret "networking-console-plugin-cert" not found
Apr 22 19:58:06.295263 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:06.295014    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-89zh2" event={"ID":"bcfcba31-705d-4865-bba0-528576e44d02","Type":"ContainerStarted","Data":"e40671900ac2e7b372a34fb27b12e95392938a21503883f5dd0a4325d80f0f87"}
Apr 22 19:58:06.298710 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:06.298657    2574 generic.go:358] "Generic (PLEG): container finished" podID="7f755d56-6d83-48c7-be8f-4efe2af27fbc" containerID="2a86e00f20cc97c39be4c8d1a74137f4a8b4bdac04871e35e182f421d664b379" exitCode=0
Apr 22 19:58:06.298849 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:06.298759    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js9v5" event={"ID":"7f755d56-6d83-48c7-be8f-4efe2af27fbc","Type":"ContainerDied","Data":"2a86e00f20cc97c39be4c8d1a74137f4a8b4bdac04871e35e182f421d664b379"}
Apr 22 19:58:06.300843 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:06.300802    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-95q85" event={"ID":"635d0f48-b24f-422c-b672-908590754a91","Type":"ContainerStarted","Data":"6642a092a049d2dfef6ebb2768d7fce15e56ce11def4dc185c8575d3c600f56e"}
Apr 22 19:58:07.306362 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:07.306323    2574 generic.go:358] "Generic (PLEG): container finished" podID="7f755d56-6d83-48c7-be8f-4efe2af27fbc" containerID="27fd2fa926b9da0716cd68067c95420ef8ac280c50c2c9138ef634f5e0f8b270" exitCode=0
Apr 22 19:58:07.306804 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:07.306381    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js9v5" event={"ID":"7f755d56-6d83-48c7-be8f-4efe2af27fbc","Type":"ContainerDied","Data":"27fd2fa926b9da0716cd68067c95420ef8ac280c50c2c9138ef634f5e0f8b270"}
Apr 22 19:58:08.312121 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:08.312071    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-js9v5" event={"ID":"7f755d56-6d83-48c7-be8f-4efe2af27fbc","Type":"ContainerStarted","Data":"5b60a49e4b58bd2da10e1e749686acb6fb693a264a33e37eb8ff566d14e11a5e"}
Apr 22 19:58:08.336841 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:08.336797    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-js9v5" podStartSLOduration=5.900312043 podStartE2EDuration="39.336779462s" podCreationTimestamp="2026-04-22 19:57:29 +0000 UTC" firstStartedPulling="2026-04-22 19:57:31.923780121 +0000 UTC m=+3.341443749" lastFinishedPulling="2026-04-22 19:58:05.360247533 +0000 UTC m=+36.777911168" observedRunningTime="2026-04-22 19:58:08.336596668 +0000 UTC m=+39.754260351" watchObservedRunningTime="2026-04-22 19:58:08.336779462 +0000 UTC m=+39.754443107"
Apr 22 19:58:09.830246 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:09.830204    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:09.830727 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:09.830258    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls\") pod \"dns-default-fs69p\" (UID: \"054ccdff-220c-47af-b943-4797d37b8723\") " pod="openshift-dns/dns-default-fs69p"
Apr 22 19:58:09.830727 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:09.830302    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-27n8r\" (UID: \"173cdd12-cd9e-403c-a2b2-e18a20be58a4\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"
Apr 22 19:58:09.830727 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:09.830340    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert\") pod \"ingress-canary-ct7n5\" (UID: \"3a7698c5-bde7-4c7c-84ac-bcdba6d86851\") " pod="openshift-ingress-canary/ingress-canary-ct7n5"
Apr 22 19:58:09.830727 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:09.830364    2574 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Apr 22 19:58:09.830727 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:09.830387    2574 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59cb695579-cvq2f: secret "image-registry-tls" not found
Apr 22 19:58:09.830727 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:09.830457    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls podName:819fb49a-5bbf-4c97-ad55-ca03015622ca nodeName:}" failed. No retries permitted until 2026-04-22 19:58:17.830436301 +0000 UTC m=+49.248099924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls") pod "image-registry-59cb695579-cvq2f" (UID: "819fb49a-5bbf-4c97-ad55-ca03015622ca") : secret "image-registry-tls" not found
Apr 22 19:58:09.830727 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:09.830460    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 22 19:58:09.830727 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:09.830513    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Apr 22 19:58:09.830727 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:09.830523    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls podName:054ccdff-220c-47af-b943-4797d37b8723 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:17.830505486 +0000 UTC m=+49.248169132 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls") pod "dns-default-fs69p" (UID: "054ccdff-220c-47af-b943-4797d37b8723") : secret "dns-default-metrics-tls" not found
Apr 22 19:58:09.830727 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:09.830464    2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 22 19:58:09.830727 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:09.830559    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert podName:173cdd12-cd9e-403c-a2b2-e18a20be58a4 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:17.830542316 +0000 UTC m=+49.248205955 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-27n8r" (UID: "173cdd12-cd9e-403c-a2b2-e18a20be58a4") : secret "networking-console-plugin-cert" not found
Apr 22 19:58:09.830727 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:09.830578    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert podName:3a7698c5-bde7-4c7c-84ac-bcdba6d86851 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:17.830568588 +0000 UTC m=+49.248232212 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert") pod "ingress-canary-ct7n5" (UID: "3a7698c5-bde7-4c7c-84ac-bcdba6d86851") : secret "canary-serving-cert" not found
Apr 22 19:58:11.318075 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:11.318040    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-95q85" event={"ID":"635d0f48-b24f-422c-b672-908590754a91","Type":"ContainerStarted","Data":"39323472996f934c0f10c404704e4b404d28cefe489d82720be2363fd8318c5c"}
Apr 22 19:58:11.318502 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:11.318104    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:58:11.319188 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:11.319168    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-89zh2" event={"ID":"bcfcba31-705d-4865-bba0-528576e44d02","Type":"ContainerStarted","Data":"0a9017ee9d28d6704afec2ce5679f98de23b2327b5c44e5ae225d0779da684bc"}
Apr 22 19:58:11.333222 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:11.333182    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-95q85" podStartSLOduration=37.067489944 podStartE2EDuration="42.333171492s" podCreationTimestamp="2026-04-22 19:57:29 +0000 UTC" firstStartedPulling="2026-04-22 19:58:05.336490968 +0000 UTC m=+36.754154599" lastFinishedPulling="2026-04-22 19:58:10.602172513 +0000 UTC m=+42.019836147" observedRunningTime="2026-04-22 19:58:11.332612985 +0000 UTC m=+42.750276624" watchObservedRunningTime="2026-04-22 19:58:11.333171492 +0000 UTC m=+42.750835137"
Apr 22 19:58:11.345339 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:11.345302    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-89zh2" podStartSLOduration=33.06968936 podStartE2EDuration="38.345291306s" podCreationTimestamp="2026-04-22 19:57:33 +0000 UTC" firstStartedPulling="2026-04-22 19:58:05.336516345 +0000 UTC m=+36.754179971" lastFinishedPulling="2026-04-22 19:58:10.612118279 +0000 UTC m=+42.029781917" observedRunningTime="2026-04-22 19:58:11.345062643 +0000 UTC m=+42.762726299" watchObservedRunningTime="2026-04-22 19:58:11.345291306 +0000 UTC m=+42.762954950"
Apr 22 19:58:17.890937 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:17.890899    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:17.890937 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:17.890942    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls\") pod \"dns-default-fs69p\" (UID: \"054ccdff-220c-47af-b943-4797d37b8723\") " pod="openshift-dns/dns-default-fs69p"
Apr 22 19:58:17.891363 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:17.890987    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-27n8r\" (UID: \"173cdd12-cd9e-403c-a2b2-e18a20be58a4\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"
Apr 22 19:58:17.891363 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:17.891022    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert\") pod \"ingress-canary-ct7n5\" (UID: \"3a7698c5-bde7-4c7c-84ac-bcdba6d86851\") " pod="openshift-ingress-canary/ingress-canary-ct7n5"
Apr 22 19:58:17.891363 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:17.891083    2574 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Apr 22 19:58:17.891363 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:17.891099    2574 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59cb695579-cvq2f: secret "image-registry-tls" not found
Apr 22 19:58:17.891363 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:17.891111    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 22 19:58:17.891363 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:17.891120    2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 22 19:58:17.891363 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:17.891133    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Apr 22 19:58:17.891363 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:17.891174    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls podName:054ccdff-220c-47af-b943-4797d37b8723 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:33.891157334 +0000 UTC m=+65.308820959 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls") pod "dns-default-fs69p" (UID: "054ccdff-220c-47af-b943-4797d37b8723") : secret "dns-default-metrics-tls" not found
Apr 22 19:58:17.891363 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:17.891188    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls podName:819fb49a-5bbf-4c97-ad55-ca03015622ca nodeName:}" failed. No retries permitted until 2026-04-22 19:58:33.891181716 +0000 UTC m=+65.308845339 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls") pod "image-registry-59cb695579-cvq2f" (UID: "819fb49a-5bbf-4c97-ad55-ca03015622ca") : secret "image-registry-tls" not found
Apr 22 19:58:17.891363 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:17.891197    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert podName:3a7698c5-bde7-4c7c-84ac-bcdba6d86851 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:33.891192783 +0000 UTC m=+65.308856406 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert") pod "ingress-canary-ct7n5" (UID: "3a7698c5-bde7-4c7c-84ac-bcdba6d86851") : secret "canary-serving-cert" not found
Apr 22 19:58:17.891363 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:17.891207    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert podName:173cdd12-cd9e-403c-a2b2-e18a20be58a4 nodeName:}" failed. No retries permitted until 2026-04-22 19:58:33.891202593 +0000 UTC m=+65.308866216 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-27n8r" (UID: "173cdd12-cd9e-403c-a2b2-e18a20be58a4") : secret "networking-console-plugin-cert" not found
Apr 22 19:58:27.284150 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:27.284125    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zlwx4"
Apr 22 19:58:33.903861 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:33.903809    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert\") pod \"ingress-canary-ct7n5\" (UID: \"3a7698c5-bde7-4c7c-84ac-bcdba6d86851\") " pod="openshift-ingress-canary/ingress-canary-ct7n5"
Apr 22 19:58:33.904301 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:33.903883    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:58:33.904301 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:33.903905    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls\") pod \"dns-default-fs69p\" (UID: \"054ccdff-220c-47af-b943-4797d37b8723\") " pod="openshift-dns/dns-default-fs69p"
Apr 22 19:58:33.904301 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:33.903936    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-27n8r\" (UID: \"173cdd12-cd9e-403c-a2b2-e18a20be58a4\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"
Apr 22 19:58:33.904301 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:33.904001    2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 22 19:58:33.904301 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:33.904025    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Apr 22 19:58:33.904301 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:33.904025    2574 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Apr 22 19:58:33.904301 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:33.904042    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 22 19:58:33.904301 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:33.904047    2574 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59cb695579-cvq2f: secret "image-registry-tls" not found
Apr 22 19:58:33.904301 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:33.904083    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert podName:3a7698c5-bde7-4c7c-84ac-bcdba6d86851 nodeName:}" failed. No retries permitted until 2026-04-22 19:59:05.904063069 +0000 UTC m=+97.321726695 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert") pod "ingress-canary-ct7n5" (UID: "3a7698c5-bde7-4c7c-84ac-bcdba6d86851") : secret "canary-serving-cert" not found
Apr 22 19:58:33.904301 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:33.904104    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls podName:819fb49a-5bbf-4c97-ad55-ca03015622ca nodeName:}" failed. No retries permitted until 2026-04-22 19:59:05.904091785 +0000 UTC m=+97.321755408 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls") pod "image-registry-59cb695579-cvq2f" (UID: "819fb49a-5bbf-4c97-ad55-ca03015622ca") : secret "image-registry-tls" not found
Apr 22 19:58:33.904301 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:33.904117    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert podName:173cdd12-cd9e-403c-a2b2-e18a20be58a4 nodeName:}" failed. No retries permitted until 2026-04-22 19:59:05.904111403 +0000 UTC m=+97.321775028 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-27n8r" (UID: "173cdd12-cd9e-403c-a2b2-e18a20be58a4") : secret "networking-console-plugin-cert" not found
Apr 22 19:58:33.904301 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:33.904127    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls podName:054ccdff-220c-47af-b943-4797d37b8723 nodeName:}" failed. No retries permitted until 2026-04-22 19:59:05.904122244 +0000 UTC m=+97.321785868 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls") pod "dns-default-fs69p" (UID: "054ccdff-220c-47af-b943-4797d37b8723") : secret "dns-default-metrics-tls" not found
Apr 22 19:58:34.810540 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:34.810507    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs\") pod \"network-metrics-daemon-9cnw2\" (UID: \"cdf7849a-206d-4b5a-ad81-e723c58f4426\") " pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:58:34.810735 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:34.810659    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Apr 22 19:58:34.810735 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:58:34.810728    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs podName:cdf7849a-206d-4b5a-ad81-e723c58f4426 nodeName:}" failed. No retries permitted until 2026-04-22 19:59:38.810710585 +0000 UTC m=+130.228374213 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs") pod "network-metrics-daemon-9cnw2" (UID: "cdf7849a-206d-4b5a-ad81-e723c58f4426") : secret "metrics-daemon-secret" not found
Apr 22 19:58:42.324078 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:58:42.324047    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-95q85"
Apr 22 19:59:05.936160 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:59:05.936016    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 19:59:05.936160 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:59:05.936077    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls\") pod \"dns-default-fs69p\" (UID: \"054ccdff-220c-47af-b943-4797d37b8723\") " pod="openshift-dns/dns-default-fs69p"
Apr 22 19:59:05.936160 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:59:05.936111    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-27n8r\" (UID: \"173cdd12-cd9e-403c-a2b2-e18a20be58a4\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"
Apr 22 19:59:05.936160 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:59:05.936137    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert\") pod \"ingress-canary-ct7n5\" (UID: \"3a7698c5-bde7-4c7c-84ac-bcdba6d86851\") " pod="openshift-ingress-canary/ingress-canary-ct7n5"
Apr 22 19:59:05.936769 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:59:05.936174    2574 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Apr 22 19:59:05.936769 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:59:05.936197    2574 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59cb695579-cvq2f: secret "image-registry-tls" not found
Apr 22 19:59:05.936769 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:59:05.936225    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Apr 22 19:59:05.936769 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:59:05.936238    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 22 19:59:05.936769 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:59:05.936270    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls podName:819fb49a-5bbf-4c97-ad55-ca03015622ca nodeName:}" failed. No retries permitted until 2026-04-22 20:00:09.93624839 +0000 UTC m=+161.353912020 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls") pod "image-registry-59cb695579-cvq2f" (UID: "819fb49a-5bbf-4c97-ad55-ca03015622ca") : secret "image-registry-tls" not found
Apr 22 19:59:05.936769 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:59:05.936288    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert podName:173cdd12-cd9e-403c-a2b2-e18a20be58a4 nodeName:}" failed. No retries permitted until 2026-04-22 20:00:09.936279303 +0000 UTC m=+161.353942932 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-27n8r" (UID: "173cdd12-cd9e-403c-a2b2-e18a20be58a4") : secret "networking-console-plugin-cert" not found
Apr 22 19:59:05.936769 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:59:05.936303    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls podName:054ccdff-220c-47af-b943-4797d37b8723 nodeName:}" failed. No retries permitted until 2026-04-22 20:00:09.936296024 +0000 UTC m=+161.353959684 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls") pod "dns-default-fs69p" (UID: "054ccdff-220c-47af-b943-4797d37b8723") : secret "dns-default-metrics-tls" not found
Apr 22 19:59:05.936769 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:59:05.936318    2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 22 19:59:05.936769 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:59:05.936364    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert podName:3a7698c5-bde7-4c7c-84ac-bcdba6d86851 nodeName:}" failed. No retries permitted until 2026-04-22 20:00:09.936355374 +0000 UTC m=+161.354018998 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert") pod "ingress-canary-ct7n5" (UID: "3a7698c5-bde7-4c7c-84ac-bcdba6d86851") : secret "canary-serving-cert" not found
Apr 22 19:59:38.866385 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:59:38.866348    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs\") pod \"network-metrics-daemon-9cnw2\" (UID: \"cdf7849a-206d-4b5a-ad81-e723c58f4426\") " pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 19:59:38.866851 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:59:38.866513    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Apr 22 19:59:38.866851 ip-10-0-135-215 kubenswrapper[2574]: E0422 19:59:38.866585    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs podName:cdf7849a-206d-4b5a-ad81-e723c58f4426 nodeName:}" failed. No retries permitted until 2026-04-22 20:01:40.86656727 +0000 UTC m=+252.284230895 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs") pod "network-metrics-daemon-9cnw2" (UID: "cdf7849a-206d-4b5a-ad81-e723c58f4426") : secret "metrics-daemon-secret" not found
Apr 22 19:59:56.632579 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:59:56.632549    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sz29m_c4edbe2f-e978-4101-8d32-e35e7e5e740e/dns-node-resolver/0.log"
Apr 22 19:59:57.632966 ip-10-0-135-215 kubenswrapper[2574]: I0422 19:59:57.632939    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vxzgj_a56d6e25-767b-43f8-b8e0-e1dd0b4879fa/node-ca/0.log"
Apr 22 20:00:04.995839 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:04.995792    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r" podUID="173cdd12-cd9e-403c-a2b2-e18a20be58a4"
Apr 22 20:00:05.004968 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:05.004938    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-59cb695579-cvq2f" podUID="819fb49a-5bbf-4c97-ad55-ca03015622ca"
Apr 22 20:00:05.015119 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:05.015094    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fs69p" podUID="054ccdff-220c-47af-b943-4797d37b8723"
Apr 22 20:00:05.020223 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:05.020199    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-ct7n5" podUID="3a7698c5-bde7-4c7c-84ac-bcdba6d86851"
Apr 22 20:00:05.174864 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:05.174837    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-9cnw2" podUID="cdf7849a-206d-4b5a-ad81-e723c58f4426"
Apr 22 20:00:05.476892 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:05.476860    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-74bb7799d9-pvlvw"]
Apr 22 20:00:05.479725 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:05.479708    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-74bb7799d9-pvlvw"
Apr 22 20:00:05.482391 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:05.482369    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Apr 22 20:00:05.483551 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:05.483529    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xd69g\""
Apr 22 20:00:05.483637 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:05.483550    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Apr 22 20:00:05.489324 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:05.489297    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-74bb7799d9-pvlvw"]
Apr 22 20:00:05.526905 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:05.526887    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 20:00:05.526994 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:05.526925    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ct7n5"
Apr 22 20:00:05.526994 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:05.526971    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fs69p"
Apr 22 20:00:05.527102 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:05.527087    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"
Apr 22 20:00:05.550672 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:05.550647    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqt5s\" (UniqueName: \"kubernetes.io/projected/45a1876b-06fe-44a6-ae0f-d15d8ad868c7-kube-api-access-qqt5s\") pod \"migrator-74bb7799d9-pvlvw\" (UID: \"45a1876b-06fe-44a6-ae0f-d15d8ad868c7\") " pod="openshift-kube-storage-version-migrator/migrator-74bb7799d9-pvlvw"
Apr 22 20:00:05.651605 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:05.651584    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qqt5s\" (UniqueName: \"kubernetes.io/projected/45a1876b-06fe-44a6-ae0f-d15d8ad868c7-kube-api-access-qqt5s\") pod \"migrator-74bb7799d9-pvlvw\" (UID: \"45a1876b-06fe-44a6-ae0f-d15d8ad868c7\") " pod="openshift-kube-storage-version-migrator/migrator-74bb7799d9-pvlvw"
Apr 22 20:00:05.660761 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:05.660738    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqt5s\" (UniqueName: \"kubernetes.io/projected/45a1876b-06fe-44a6-ae0f-d15d8ad868c7-kube-api-access-qqt5s\") pod \"migrator-74bb7799d9-pvlvw\" (UID: \"45a1876b-06fe-44a6-ae0f-d15d8ad868c7\") " pod="openshift-kube-storage-version-migrator/migrator-74bb7799d9-pvlvw"
Apr 22 20:00:05.788443 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:05.788357    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-74bb7799d9-pvlvw"
Apr 22 20:00:05.898696 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:05.898667    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-74bb7799d9-pvlvw"]
Apr 22 20:00:05.901700 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:00:05.901669    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a1876b_06fe_44a6_ae0f_d15d8ad868c7.slice/crio-8efb1f9e3eb7b7a5840a3be94eee7d2bb04c21155e808a5eff3aa7ff1ba3405f WatchSource:0}: Error finding container 8efb1f9e3eb7b7a5840a3be94eee7d2bb04c21155e808a5eff3aa7ff1ba3405f: Status 404 returned error can't find the container with id 8efb1f9e3eb7b7a5840a3be94eee7d2bb04c21155e808a5eff3aa7ff1ba3405f
Apr 22 20:00:06.530299 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:06.530218    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-74bb7799d9-pvlvw" event={"ID":"45a1876b-06fe-44a6-ae0f-d15d8ad868c7","Type":"ContainerStarted","Data":"8efb1f9e3eb7b7a5840a3be94eee7d2bb04c21155e808a5eff3aa7ff1ba3405f"}
Apr 22 20:00:07.533810 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:07.533775    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-74bb7799d9-pvlvw" event={"ID":"45a1876b-06fe-44a6-ae0f-d15d8ad868c7","Type":"ContainerStarted","Data":"21ec60405a54d4c1c99201b363a09c306fd98f70998c9887d5cde0e52c8c0cc3"}
Apr 22 20:00:07.534152 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:07.533814    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-74bb7799d9-pvlvw" event={"ID":"45a1876b-06fe-44a6-ae0f-d15d8ad868c7","Type":"ContainerStarted","Data":"33b7aec3c9f061340cd11a6d2aec9dc974812bc72c606d601d52f6a12cc1674f"}
Apr 22 20:00:07.548313 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:07.548270    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-74bb7799d9-pvlvw" podStartSLOduration=1.483582242 podStartE2EDuration="2.548256492s" podCreationTimestamp="2026-04-22 20:00:05 +0000 UTC" firstStartedPulling="2026-04-22 20:00:05.903480896 +0000 UTC m=+157.321144520" lastFinishedPulling="2026-04-22 20:00:06.968155126 +0000 UTC m=+158.385818770" observedRunningTime="2026-04-22 20:00:07.547989849 +0000 UTC m=+158.965653507" watchObservedRunningTime="2026-04-22 20:00:07.548256492 +0000 UTC m=+158.965920156"
Apr 22 20:00:07.817331 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:07.817268    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-865cb79987-sk86x"]
Apr 22 20:00:07.820052 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:07.820037    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-865cb79987-sk86x"
Apr 22 20:00:07.822638 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:07.822614    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Apr 22 20:00:07.822638 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:07.822628    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-2zcx6\""
Apr 22 20:00:07.822768 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:07.822619    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Apr 22 20:00:07.822768 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:07.822666    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Apr 22 20:00:07.823673 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:07.823659    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Apr 22 20:00:07.826754 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:07.826731    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-865cb79987-sk86x"]
Apr 22 20:00:07.867041 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:07.867014    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/aa9ea456-3213-4efc-891a-73fe4e18fb3e-signing-cabundle\") pod \"service-ca-865cb79987-sk86x\" (UID: \"aa9ea456-3213-4efc-891a-73fe4e18fb3e\") " pod="openshift-service-ca/service-ca-865cb79987-sk86x"
Apr 22 20:00:07.867148 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:07.867059    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7wkj\" (UniqueName: \"kubernetes.io/projected/aa9ea456-3213-4efc-891a-73fe4e18fb3e-kube-api-access-f7wkj\") pod \"service-ca-865cb79987-sk86x\" (UID: \"aa9ea456-3213-4efc-891a-73fe4e18fb3e\") " pod="openshift-service-ca/service-ca-865cb79987-sk86x"
Apr 22 20:00:07.867192 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:07.867142    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/aa9ea456-3213-4efc-891a-73fe4e18fb3e-signing-key\") pod \"service-ca-865cb79987-sk86x\" (UID: \"aa9ea456-3213-4efc-891a-73fe4e18fb3e\") " pod="openshift-service-ca/service-ca-865cb79987-sk86x"
Apr 22 20:00:07.967688 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:07.967665    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/aa9ea456-3213-4efc-891a-73fe4e18fb3e-signing-key\") pod \"service-ca-865cb79987-sk86x\" (UID: \"aa9ea456-3213-4efc-891a-73fe4e18fb3e\") " pod="openshift-service-ca/service-ca-865cb79987-sk86x"
Apr 22 20:00:07.967810 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:07.967709    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/aa9ea456-3213-4efc-891a-73fe4e18fb3e-signing-cabundle\") pod \"service-ca-865cb79987-sk86x\" (UID: \"aa9ea456-3213-4efc-891a-73fe4e18fb3e\") " pod="openshift-service-ca/service-ca-865cb79987-sk86x"
Apr 22 20:00:07.967810 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:07.967735    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f7wkj\" (UniqueName: \"kubernetes.io/projected/aa9ea456-3213-4efc-891a-73fe4e18fb3e-kube-api-access-f7wkj\") pod \"service-ca-865cb79987-sk86x\" (UID: \"aa9ea456-3213-4efc-891a-73fe4e18fb3e\") " pod="openshift-service-ca/service-ca-865cb79987-sk86x"
Apr 22 20:00:07.968382 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:07.968365    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/aa9ea456-3213-4efc-891a-73fe4e18fb3e-signing-cabundle\") pod \"service-ca-865cb79987-sk86x\" (UID: \"aa9ea456-3213-4efc-891a-73fe4e18fb3e\") " pod="openshift-service-ca/service-ca-865cb79987-sk86x"
Apr 22 20:00:07.970212 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:07.970191    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/aa9ea456-3213-4efc-891a-73fe4e18fb3e-signing-key\") pod \"service-ca-865cb79987-sk86x\" (UID: \"aa9ea456-3213-4efc-891a-73fe4e18fb3e\") " pod="openshift-service-ca/service-ca-865cb79987-sk86x"
Apr 22 20:00:07.976171 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:07.976145    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7wkj\" (UniqueName: \"kubernetes.io/projected/aa9ea456-3213-4efc-891a-73fe4e18fb3e-kube-api-access-f7wkj\") pod \"service-ca-865cb79987-sk86x\" (UID: \"aa9ea456-3213-4efc-891a-73fe4e18fb3e\") " pod="openshift-service-ca/service-ca-865cb79987-sk86x"
Apr 22 20:00:08.102066 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.102011    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kqn98"]
Apr 22 20:00:08.105280 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.105266    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:08.107888 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.107866    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Apr 22 20:00:08.107993 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.107977    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Apr 22 20:00:08.108051 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.108040    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Apr 22 20:00:08.108220 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.108195    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Apr 22 20:00:08.108331 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.108238    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5prhs\""
Apr 22 20:00:08.116096 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.116077    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kqn98"]
Apr 22 20:00:08.129370 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.129351    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-865cb79987-sk86x"
Apr 22 20:00:08.169582 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.169551    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb877699-2e26-41f4-8c95-f866532b38ae-crio-socket\") pod \"insights-runtime-extractor-kqn98\" (UID: \"eb877699-2e26-41f4-8c95-f866532b38ae\") " pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:08.169694 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.169604    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb877699-2e26-41f4-8c95-f866532b38ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kqn98\" (UID: \"eb877699-2e26-41f4-8c95-f866532b38ae\") " pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:08.169694 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.169666    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9qhf\" (UniqueName: \"kubernetes.io/projected/eb877699-2e26-41f4-8c95-f866532b38ae-kube-api-access-x9qhf\") pod \"insights-runtime-extractor-kqn98\" (UID: \"eb877699-2e26-41f4-8c95-f866532b38ae\") " pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:08.169795 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.169706    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb877699-2e26-41f4-8c95-f866532b38ae-data-volume\") pod \"insights-runtime-extractor-kqn98\" (UID: \"eb877699-2e26-41f4-8c95-f866532b38ae\") " pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:08.169795 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.169781    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb877699-2e26-41f4-8c95-f866532b38ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kqn98\" (UID: \"eb877699-2e26-41f4-8c95-f866532b38ae\") " pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:08.240336 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.240315    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-865cb79987-sk86x"]
Apr 22 20:00:08.242467 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:00:08.242433    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9ea456_3213_4efc_891a_73fe4e18fb3e.slice/crio-195b76f398d3a823b51fb0c55a52ddca1839745a09b695cffcff168ef7f0828b WatchSource:0}: Error finding container 195b76f398d3a823b51fb0c55a52ddca1839745a09b695cffcff168ef7f0828b: Status 404 returned error can't find the container with id 195b76f398d3a823b51fb0c55a52ddca1839745a09b695cffcff168ef7f0828b
Apr 22 20:00:08.271152 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.271132    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb877699-2e26-41f4-8c95-f866532b38ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kqn98\" (UID: \"eb877699-2e26-41f4-8c95-f866532b38ae\") " pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:08.271254 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.271177    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb877699-2e26-41f4-8c95-f866532b38ae-crio-socket\") pod \"insights-runtime-extractor-kqn98\" (UID: \"eb877699-2e26-41f4-8c95-f866532b38ae\") " pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:08.271254 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.271209    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb877699-2e26-41f4-8c95-f866532b38ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kqn98\" (UID: \"eb877699-2e26-41f4-8c95-f866532b38ae\") " pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:08.271254 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.271239    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x9qhf\" (UniqueName: \"kubernetes.io/projected/eb877699-2e26-41f4-8c95-f866532b38ae-kube-api-access-x9qhf\") pod \"insights-runtime-extractor-kqn98\" (UID: \"eb877699-2e26-41f4-8c95-f866532b38ae\") " pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:08.271401 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.271271    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb877699-2e26-41f4-8c95-f866532b38ae-data-volume\") pod \"insights-runtime-extractor-kqn98\" (UID: \"eb877699-2e26-41f4-8c95-f866532b38ae\") " pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:08.271401 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.271278    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb877699-2e26-41f4-8c95-f866532b38ae-crio-socket\") pod \"insights-runtime-extractor-kqn98\" (UID: \"eb877699-2e26-41f4-8c95-f866532b38ae\") " pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:08.271401 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:08.271390    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Apr 22 20:00:08.271577 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:08.271482    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb877699-2e26-41f4-8c95-f866532b38ae-insights-runtime-extractor-tls podName:eb877699-2e26-41f4-8c95-f866532b38ae nodeName:}" failed. No retries permitted until 2026-04-22 20:00:08.771461357 +0000 UTC m=+160.189124982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb877699-2e26-41f4-8c95-f866532b38ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kqn98" (UID: "eb877699-2e26-41f4-8c95-f866532b38ae") : secret "insights-runtime-extractor-tls" not found
Apr 22 20:00:08.271679 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.271660    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb877699-2e26-41f4-8c95-f866532b38ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kqn98\" (UID: \"eb877699-2e26-41f4-8c95-f866532b38ae\") " pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:08.272317 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.272297    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb877699-2e26-41f4-8c95-f866532b38ae-data-volume\") pod \"insights-runtime-extractor-kqn98\" (UID: \"eb877699-2e26-41f4-8c95-f866532b38ae\") " pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:08.281809 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.281789    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9qhf\" (UniqueName: \"kubernetes.io/projected/eb877699-2e26-41f4-8c95-f866532b38ae-kube-api-access-x9qhf\") pod \"insights-runtime-extractor-kqn98\" (UID: \"eb877699-2e26-41f4-8c95-f866532b38ae\") " pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:08.536584 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.536557    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-865cb79987-sk86x" event={"ID":"aa9ea456-3213-4efc-891a-73fe4e18fb3e","Type":"ContainerStarted","Data":"195b76f398d3a823b51fb0c55a52ddca1839745a09b695cffcff168ef7f0828b"}
Apr 22 20:00:08.775951 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:08.775918    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb877699-2e26-41f4-8c95-f866532b38ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kqn98\" (UID: \"eb877699-2e26-41f4-8c95-f866532b38ae\") " pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:08.776089 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:08.776032    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Apr 22 20:00:08.776142 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:08.776097    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb877699-2e26-41f4-8c95-f866532b38ae-insights-runtime-extractor-tls podName:eb877699-2e26-41f4-8c95-f866532b38ae nodeName:}" failed. No retries permitted until 2026-04-22 20:00:09.776082705 +0000 UTC m=+161.193746329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb877699-2e26-41f4-8c95-f866532b38ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kqn98" (UID: "eb877699-2e26-41f4-8c95-f866532b38ae") : secret "insights-runtime-extractor-tls" not found
Apr 22 20:00:09.786146 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:09.786104    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb877699-2e26-41f4-8c95-f866532b38ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kqn98\" (UID: \"eb877699-2e26-41f4-8c95-f866532b38ae\") " pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:09.786662 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:09.786247    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Apr 22 20:00:09.786662 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:09.786326    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb877699-2e26-41f4-8c95-f866532b38ae-insights-runtime-extractor-tls podName:eb877699-2e26-41f4-8c95-f866532b38ae nodeName:}" failed. No retries permitted until 2026-04-22 20:00:11.786302599 +0000 UTC m=+163.203966226 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb877699-2e26-41f4-8c95-f866532b38ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kqn98" (UID: "eb877699-2e26-41f4-8c95-f866532b38ae") : secret "insights-runtime-extractor-tls" not found
Apr 22 20:00:09.988054 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:09.988022    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls\") pod \"dns-default-fs69p\" (UID: \"054ccdff-220c-47af-b943-4797d37b8723\") " pod="openshift-dns/dns-default-fs69p"
Apr 22 20:00:09.988191 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:09.988085    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-27n8r\" (UID: \"173cdd12-cd9e-403c-a2b2-e18a20be58a4\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"
Apr 22 20:00:09.988191 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:09.988124    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert\") pod \"ingress-canary-ct7n5\" (UID: \"3a7698c5-bde7-4c7c-84ac-bcdba6d86851\") " pod="openshift-ingress-canary/ingress-canary-ct7n5"
Apr 22 20:00:09.988191 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:09.988168    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Apr 22 20:00:09.988498 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:09.988191    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls\") pod \"image-registry-59cb695579-cvq2f\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") " pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 20:00:09.988498 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:09.988209    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Apr 22 20:00:09.988498 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:09.988232    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls podName:054ccdff-220c-47af-b943-4797d37b8723 nodeName:}" failed. No retries permitted until 2026-04-22 20:02:11.988212175 +0000 UTC m=+283.405875816 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls") pod "dns-default-fs69p" (UID: "054ccdff-220c-47af-b943-4797d37b8723") : secret "dns-default-metrics-tls" not found
Apr 22 20:00:09.988498 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:09.988249    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert podName:173cdd12-cd9e-403c-a2b2-e18a20be58a4 nodeName:}" failed. No retries permitted until 2026-04-22 20:02:11.988238346 +0000 UTC m=+283.405901971 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-27n8r" (UID: "173cdd12-cd9e-403c-a2b2-e18a20be58a4") : secret "networking-console-plugin-cert" not found
Apr 22 20:00:09.988498 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:09.988306    2574 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Apr 22 20:00:09.988498 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:09.988365    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert podName:3a7698c5-bde7-4c7c-84ac-bcdba6d86851 nodeName:}" failed. No retries permitted until 2026-04-22 20:02:11.988349792 +0000 UTC m=+283.406013421 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert") pod "ingress-canary-ct7n5" (UID: "3a7698c5-bde7-4c7c-84ac-bcdba6d86851") : secret "canary-serving-cert" not found
Apr 22 20:00:09.988498 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:09.988308    2574 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Apr 22 20:00:09.988498 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:09.988388    2574 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59cb695579-cvq2f: secret "image-registry-tls" not found
Apr 22 20:00:09.988498 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:09.988440    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls podName:819fb49a-5bbf-4c97-ad55-ca03015622ca nodeName:}" failed. No retries permitted until 2026-04-22 20:02:11.988429611 +0000 UTC m=+283.406093241 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls") pod "image-registry-59cb695579-cvq2f" (UID: "819fb49a-5bbf-4c97-ad55-ca03015622ca") : secret "image-registry-tls" not found
Apr 22 20:00:10.542221 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:10.542188    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-865cb79987-sk86x" event={"ID":"aa9ea456-3213-4efc-891a-73fe4e18fb3e","Type":"ContainerStarted","Data":"3e9097d59f728f2af908c11a0f6f15f717d33e67174948921ec9d81a9ae7949a"}
Apr 22 20:00:10.557539 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:10.557495    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-865cb79987-sk86x" podStartSLOduration=1.922584947 podStartE2EDuration="3.557479467s" podCreationTimestamp="2026-04-22 20:00:07 +0000 UTC" firstStartedPulling="2026-04-22 20:00:08.244300765 +0000 UTC m=+159.661964389" lastFinishedPulling="2026-04-22 20:00:09.879195279 +0000 UTC m=+161.296858909" observedRunningTime="2026-04-22 20:00:10.556890868 +0000 UTC m=+161.974554511" watchObservedRunningTime="2026-04-22 20:00:10.557479467 +0000 UTC m=+161.975143115"
Apr 22 20:00:11.801989 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:11.801941    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb877699-2e26-41f4-8c95-f866532b38ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kqn98\" (UID: \"eb877699-2e26-41f4-8c95-f866532b38ae\") " pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:11.802453 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:11.802080    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Apr 22 20:00:11.802453 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:11.802140    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb877699-2e26-41f4-8c95-f866532b38ae-insights-runtime-extractor-tls podName:eb877699-2e26-41f4-8c95-f866532b38ae nodeName:}" failed. No retries permitted until 2026-04-22 20:00:15.80212176 +0000 UTC m=+167.219785389 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb877699-2e26-41f4-8c95-f866532b38ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kqn98" (UID: "eb877699-2e26-41f4-8c95-f866532b38ae") : secret "insights-runtime-extractor-tls" not found
Apr 22 20:00:15.829956 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:15.829915    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb877699-2e26-41f4-8c95-f866532b38ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kqn98\" (UID: \"eb877699-2e26-41f4-8c95-f866532b38ae\") " pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:15.830366 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:15.830053    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Apr 22 20:00:15.830366 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:15.830127    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb877699-2e26-41f4-8c95-f866532b38ae-insights-runtime-extractor-tls podName:eb877699-2e26-41f4-8c95-f866532b38ae nodeName:}" failed. No retries permitted until 2026-04-22 20:00:23.830111533 +0000 UTC m=+175.247775156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb877699-2e26-41f4-8c95-f866532b38ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kqn98" (UID: "eb877699-2e26-41f4-8c95-f866532b38ae") : secret "insights-runtime-extractor-tls" not found
Apr 22 20:00:19.158019 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:19.157942    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 20:00:23.884253 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:23.884208    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb877699-2e26-41f4-8c95-f866532b38ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kqn98\" (UID: \"eb877699-2e26-41f4-8c95-f866532b38ae\") " pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:23.886631 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:23.886605    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb877699-2e26-41f4-8c95-f866532b38ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kqn98\" (UID: \"eb877699-2e26-41f4-8c95-f866532b38ae\") " pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:24.014269 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:24.014241    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kqn98"
Apr 22 20:00:24.125735 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:24.125705    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kqn98"]
Apr 22 20:00:24.128517 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:00:24.128487    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb877699_2e26_41f4_8c95_f866532b38ae.slice/crio-06edfa9385b9db7dcf34e1b419b87a54ac62edb076dbe57961d751c1d904c514 WatchSource:0}: Error finding container 06edfa9385b9db7dcf34e1b419b87a54ac62edb076dbe57961d751c1d904c514: Status 404 returned error can't find the container with id 06edfa9385b9db7dcf34e1b419b87a54ac62edb076dbe57961d751c1d904c514
Apr 22 20:00:24.570490 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:24.570446    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kqn98" event={"ID":"eb877699-2e26-41f4-8c95-f866532b38ae","Type":"ContainerStarted","Data":"8516c908b6f836dfd2ecb28ff54b4f8088afc20a66ff37d318ad1b22df4d74a2"}
Apr 22 20:00:24.570490 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:24.570488    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kqn98" event={"ID":"eb877699-2e26-41f4-8c95-f866532b38ae","Type":"ContainerStarted","Data":"06edfa9385b9db7dcf34e1b419b87a54ac62edb076dbe57961d751c1d904c514"}
Apr 22 20:00:25.575073 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:25.575034    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kqn98" event={"ID":"eb877699-2e26-41f4-8c95-f866532b38ae","Type":"ContainerStarted","Data":"2fae8b12817987caa886f1f936c1c61157c4e3185fe7808dd614b35f3fbf8db6"}
Apr 22 20:00:26.348974 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:26.348950    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-rcmm5"]
Apr 22 20:00:26.351766 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:26.351745    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-rcmm5"
Apr 22 20:00:26.354663 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:26.354639    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-v66k9\""
Apr 22 20:00:26.354663 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:26.354651    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Apr 22 20:00:26.362488 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:26.362468    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-rcmm5"]
Apr 22 20:00:26.504244 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:26.504208    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/88ede3f5-5ebd-4226-94d9-6a2b8687ebe6-tls-certificates\") pod \"prometheus-operator-admission-webhook-57cf98b594-rcmm5\" (UID: \"88ede3f5-5ebd-4226-94d9-6a2b8687ebe6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-rcmm5"
Apr 22 20:00:26.579851 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:26.579773    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kqn98" event={"ID":"eb877699-2e26-41f4-8c95-f866532b38ae","Type":"ContainerStarted","Data":"25b9c9b2f920337f293345368d906788d0bbd53b02c471ed8302cbd93b4d7b62"}
Apr 22 20:00:26.601652 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:26.601599    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kqn98" podStartSLOduration=16.504898877 podStartE2EDuration="18.601585131s" podCreationTimestamp="2026-04-22 20:00:08 +0000 UTC" firstStartedPulling="2026-04-22 20:00:24.183075469 +0000 UTC m=+175.600739097" lastFinishedPulling="2026-04-22 20:00:26.279761725 +0000 UTC m=+177.697425351" observedRunningTime="2026-04-22 20:00:26.600640354 +0000 UTC m=+178.018304001" watchObservedRunningTime="2026-04-22 20:00:26.601585131 +0000 UTC m=+178.019248777"
Apr 22 20:00:26.605263 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:26.605236    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/88ede3f5-5ebd-4226-94d9-6a2b8687ebe6-tls-certificates\") pod \"prometheus-operator-admission-webhook-57cf98b594-rcmm5\" (UID: \"88ede3f5-5ebd-4226-94d9-6a2b8687ebe6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-rcmm5"
Apr 22 20:00:26.607716 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:26.607691    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/88ede3f5-5ebd-4226-94d9-6a2b8687ebe6-tls-certificates\") pod \"prometheus-operator-admission-webhook-57cf98b594-rcmm5\" (UID: \"88ede3f5-5ebd-4226-94d9-6a2b8687ebe6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-rcmm5"
Apr 22 20:00:26.659652 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:26.659625    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-rcmm5"
Apr 22 20:00:26.768890 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:26.768869    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-rcmm5"]
Apr 22 20:00:26.771190 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:00:26.771165    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ede3f5_5ebd_4226_94d9_6a2b8687ebe6.slice/crio-7c68dc0cdaa9d215aac71cd207d98aa8aab69dc3699fc28a207ac20633fde288 WatchSource:0}: Error finding container 7c68dc0cdaa9d215aac71cd207d98aa8aab69dc3699fc28a207ac20633fde288: Status 404 returned error can't find the container with id 7c68dc0cdaa9d215aac71cd207d98aa8aab69dc3699fc28a207ac20633fde288
Apr 22 20:00:27.583120 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:27.583086    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-rcmm5" event={"ID":"88ede3f5-5ebd-4226-94d9-6a2b8687ebe6","Type":"ContainerStarted","Data":"7c68dc0cdaa9d215aac71cd207d98aa8aab69dc3699fc28a207ac20633fde288"}
Apr 22 20:00:28.586995 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:28.586962    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-rcmm5" event={"ID":"88ede3f5-5ebd-4226-94d9-6a2b8687ebe6","Type":"ContainerStarted","Data":"61f911134b538abd3210b5bb9b4ddb60f1ed714e49a742ed50233133d8cdbb8a"}
Apr 22 20:00:28.587336 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:28.587176    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-rcmm5"
Apr 22 20:00:28.592621 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:28.592594    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-rcmm5"
Apr 22 20:00:28.603091 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:28.603042    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-rcmm5" podStartSLOduration=1.562110967 podStartE2EDuration="2.603019005s" podCreationTimestamp="2026-04-22 20:00:26 +0000 UTC" firstStartedPulling="2026-04-22 20:00:26.773545747 +0000 UTC m=+178.191209371" lastFinishedPulling="2026-04-22 20:00:27.814453785 +0000 UTC m=+179.232117409" observedRunningTime="2026-04-22 20:00:28.602429919 +0000 UTC m=+180.020093566" watchObservedRunningTime="2026-04-22 20:00:28.603019005 +0000 UTC m=+180.020682653"
Apr 22 20:00:33.779318 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.779284    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"]
Apr 22 20:00:33.783229 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.783201    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"
Apr 22 20:00:33.785304 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.785279    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mbktf"]
Apr 22 20:00:33.787253 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.787227    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Apr 22 20:00:33.787387 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.787266    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Apr 22 20:00:33.788161 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.788143    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Apr 22 20:00:33.788447 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.788227    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Apr 22 20:00:33.788604 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.788322    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qgrdx\""
Apr 22 20:00:33.788604 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.788598    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.788732 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.788329    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Apr 22 20:00:33.788862 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.788388    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Apr 22 20:00:33.793182 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.793145    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Apr 22 20:00:33.793394 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.793375    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vfgt6\""
Apr 22 20:00:33.793763 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.793746    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Apr 22 20:00:33.794668 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.794645    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"]
Apr 22 20:00:33.794759 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.794741    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Apr 22 20:00:33.859129 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.859093    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0aefc85e-890b-4111-b151-cdea5415289d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-69db897b98-s4j8t\" (UID: \"0aefc85e-890b-4111-b151-cdea5415289d\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"
Apr 22 20:00:33.859291 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.859168    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-sys\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.859291 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.859193    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-node-exporter-accelerators-collector-config\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.859291 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.859226    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-node-exporter-tls\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.859481 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.859301    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-root\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.859481 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.859346    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-node-exporter-wtmp\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.859481 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.859409    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0aefc85e-890b-4111-b151-cdea5415289d-kube-state-metrics-tls\") pod \"kube-state-metrics-69db897b98-s4j8t\" (UID: \"0aefc85e-890b-4111-b151-cdea5415289d\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"
Apr 22 20:00:33.859481 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.859457    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.859629 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.859508    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdmtf\" (UniqueName: \"kubernetes.io/projected/0aefc85e-890b-4111-b151-cdea5415289d-kube-api-access-qdmtf\") pod \"kube-state-metrics-69db897b98-s4j8t\" (UID: \"0aefc85e-890b-4111-b151-cdea5415289d\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"
Apr 22 20:00:33.859629 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.859538    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-node-exporter-textfile\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.859629 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.859569    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0aefc85e-890b-4111-b151-cdea5415289d-metrics-client-ca\") pod \"kube-state-metrics-69db897b98-s4j8t\" (UID: \"0aefc85e-890b-4111-b151-cdea5415289d\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"
Apr 22 20:00:33.859629 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.859595    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86j88\" (UniqueName: \"kubernetes.io/projected/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-kube-api-access-86j88\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.859817 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.859639    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0aefc85e-890b-4111-b151-cdea5415289d-volume-directive-shadow\") pod \"kube-state-metrics-69db897b98-s4j8t\" (UID: \"0aefc85e-890b-4111-b151-cdea5415289d\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"
Apr 22 20:00:33.859817 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.859664    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-metrics-client-ca\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.859817 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.859715    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0aefc85e-890b-4111-b151-cdea5415289d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-69db897b98-s4j8t\" (UID: \"0aefc85e-890b-4111-b151-cdea5415289d\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"
Apr 22 20:00:33.961160 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.961125    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0aefc85e-890b-4111-b151-cdea5415289d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-69db897b98-s4j8t\" (UID: \"0aefc85e-890b-4111-b151-cdea5415289d\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"
Apr 22 20:00:33.961332 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.961181    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-sys\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.961332 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.961209    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-node-exporter-accelerators-collector-config\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.961332 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.961238    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-node-exporter-tls\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.961332 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.961268    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-root\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.961332 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.961297    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-node-exporter-wtmp\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.961332 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.961307    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-sys\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.961665 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.961341    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0aefc85e-890b-4111-b151-cdea5415289d-kube-state-metrics-tls\") pod \"kube-state-metrics-69db897b98-s4j8t\" (UID: \"0aefc85e-890b-4111-b151-cdea5415289d\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"
Apr 22 20:00:33.961665 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.961351    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-root\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.961665 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.961366    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.961665 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.961396    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qdmtf\" (UniqueName: \"kubernetes.io/projected/0aefc85e-890b-4111-b151-cdea5415289d-kube-api-access-qdmtf\") pod \"kube-state-metrics-69db897b98-s4j8t\" (UID: \"0aefc85e-890b-4111-b151-cdea5415289d\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"
Apr 22 20:00:33.961665 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.961440    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-node-exporter-textfile\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.961665 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.961467    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-node-exporter-wtmp\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.961665 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.961472    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0aefc85e-890b-4111-b151-cdea5415289d-metrics-client-ca\") pod \"kube-state-metrics-69db897b98-s4j8t\" (UID: \"0aefc85e-890b-4111-b151-cdea5415289d\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"
Apr 22 20:00:33.961665 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.961513    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-86j88\" (UniqueName: \"kubernetes.io/projected/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-kube-api-access-86j88\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.961665 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.961563    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0aefc85e-890b-4111-b151-cdea5415289d-volume-directive-shadow\") pod \"kube-state-metrics-69db897b98-s4j8t\" (UID: \"0aefc85e-890b-4111-b151-cdea5415289d\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"
Apr 22 20:00:33.961665 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.961593    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-metrics-client-ca\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.961665 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.961622    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0aefc85e-890b-4111-b151-cdea5415289d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-69db897b98-s4j8t\" (UID: \"0aefc85e-890b-4111-b151-cdea5415289d\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"
Apr 22 20:00:33.962172 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.961858    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-node-exporter-accelerators-collector-config\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.962172 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:33.962061    2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Apr 22 20:00:33.962172 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:33.962119    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-node-exporter-tls podName:40ed9426-93f6-4d39-af17-0aff9cbfc8b7 nodeName:}" failed. No retries permitted until 2026-04-22 20:00:34.462099995 +0000 UTC m=+185.879763620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-node-exporter-tls") pod "node-exporter-mbktf" (UID: "40ed9426-93f6-4d39-af17-0aff9cbfc8b7") : secret "node-exporter-tls" not found
Apr 22 20:00:33.962172 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.962124    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0aefc85e-890b-4111-b151-cdea5415289d-metrics-client-ca\") pod \"kube-state-metrics-69db897b98-s4j8t\" (UID: \"0aefc85e-890b-4111-b151-cdea5415289d\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"
Apr 22 20:00:33.962394 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.962290    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-metrics-client-ca\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.962394 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.962382    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0aefc85e-890b-4111-b151-cdea5415289d-volume-directive-shadow\") pod \"kube-state-metrics-69db897b98-s4j8t\" (UID: \"0aefc85e-890b-4111-b151-cdea5415289d\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"
Apr 22 20:00:33.962522 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.962440    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0aefc85e-890b-4111-b151-cdea5415289d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-69db897b98-s4j8t\" (UID: \"0aefc85e-890b-4111-b151-cdea5415289d\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"
Apr 22 20:00:33.962853 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.962824    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-node-exporter-textfile\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.964864 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.964838    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.964864 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.964854    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0aefc85e-890b-4111-b151-cdea5415289d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-69db897b98-s4j8t\" (UID: \"0aefc85e-890b-4111-b151-cdea5415289d\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"
Apr 22 20:00:33.965019 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.964881    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0aefc85e-890b-4111-b151-cdea5415289d-kube-state-metrics-tls\") pod \"kube-state-metrics-69db897b98-s4j8t\" (UID: \"0aefc85e-890b-4111-b151-cdea5415289d\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"
Apr 22 20:00:33.970910 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.970888    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86j88\" (UniqueName: \"kubernetes.io/projected/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-kube-api-access-86j88\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:33.971025 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:33.970922    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdmtf\" (UniqueName: \"kubernetes.io/projected/0aefc85e-890b-4111-b151-cdea5415289d-kube-api-access-qdmtf\") pod \"kube-state-metrics-69db897b98-s4j8t\" (UID: \"0aefc85e-890b-4111-b151-cdea5415289d\") " pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"
Apr 22 20:00:34.100486 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:34.100371    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"
Apr 22 20:00:34.221564 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:34.221540    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-69db897b98-s4j8t"]
Apr 22 20:00:34.224058 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:00:34.224031    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aefc85e_890b_4111_b151_cdea5415289d.slice/crio-d45111b56be213d4d14d5d09013d8e5535d5d88e427addf73837867467f455e6 WatchSource:0}: Error finding container d45111b56be213d4d14d5d09013d8e5535d5d88e427addf73837867467f455e6: Status 404 returned error can't find the container with id d45111b56be213d4d14d5d09013d8e5535d5d88e427addf73837867467f455e6
Apr 22 20:00:34.465774 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:34.465737    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-node-exporter-tls\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:34.468703 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:34.468679    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/40ed9426-93f6-4d39-af17-0aff9cbfc8b7-node-exporter-tls\") pod \"node-exporter-mbktf\" (UID: \"40ed9426-93f6-4d39-af17-0aff9cbfc8b7\") " pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:34.601440 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:34.601389    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t" event={"ID":"0aefc85e-890b-4111-b151-cdea5415289d","Type":"ContainerStarted","Data":"d45111b56be213d4d14d5d09013d8e5535d5d88e427addf73837867467f455e6"}
Apr 22 20:00:34.706118 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:34.706088    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mbktf"
Apr 22 20:00:34.713496 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:00:34.713461    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ed9426_93f6_4d39_af17_0aff9cbfc8b7.slice/crio-8839fcd0711e8ed20bb3918621584f8167f104f9bf6cadef6aa359b0a8597452 WatchSource:0}: Error finding container 8839fcd0711e8ed20bb3918621584f8167f104f9bf6cadef6aa359b0a8597452: Status 404 returned error can't find the container with id 8839fcd0711e8ed20bb3918621584f8167f104f9bf6cadef6aa359b0a8597452
Apr 22 20:00:35.605551 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:35.605517    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mbktf" event={"ID":"40ed9426-93f6-4d39-af17-0aff9cbfc8b7","Type":"ContainerStarted","Data":"8839fcd0711e8ed20bb3918621584f8167f104f9bf6cadef6aa359b0a8597452"}
Apr 22 20:00:36.609975 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:36.609940    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t" event={"ID":"0aefc85e-890b-4111-b151-cdea5415289d","Type":"ContainerStarted","Data":"75486cf47d8ff2fdde981429392f5b6f10ce702aa14bb2f9a10bd61b28ccc7bb"}
Apr 22 20:00:36.609975 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:36.609977    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t" event={"ID":"0aefc85e-890b-4111-b151-cdea5415289d","Type":"ContainerStarted","Data":"d693f22a77d2238b10bdcbc812be03ae08810c7febaf58632d662a7872a3b618"}
Apr 22 20:00:36.610456 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:36.609989    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t" event={"ID":"0aefc85e-890b-4111-b151-cdea5415289d","Type":"ContainerStarted","Data":"74a971e0a46efaefd2cee176a8db844f3734875226ee7987f6d2547b6f383bb6"}
Apr 22 20:00:36.611451 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:36.611405    2574 generic.go:358] "Generic (PLEG): container finished" podID="40ed9426-93f6-4d39-af17-0aff9cbfc8b7" containerID="d929a5a1857425d09b704a71ee1c29aaed97c97e5006d2b7c8fdae4528ee3334" exitCode=0
Apr 22 20:00:36.611560 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:36.611496    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mbktf" event={"ID":"40ed9426-93f6-4d39-af17-0aff9cbfc8b7","Type":"ContainerDied","Data":"d929a5a1857425d09b704a71ee1c29aaed97c97e5006d2b7c8fdae4528ee3334"}
Apr 22 20:00:36.630469 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:36.630404    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-69db897b98-s4j8t" podStartSLOduration=2.294790021 podStartE2EDuration="3.630390606s" podCreationTimestamp="2026-04-22 20:00:33 +0000 UTC" firstStartedPulling="2026-04-22 20:00:34.225868166 +0000 UTC m=+185.643531790" lastFinishedPulling="2026-04-22 20:00:35.56146873 +0000 UTC m=+186.979132375" observedRunningTime="2026-04-22 20:00:36.629333464 +0000 UTC m=+188.046997103" watchObservedRunningTime="2026-04-22 20:00:36.630390606 +0000 UTC m=+188.048054251"
Apr 22 20:00:37.616069 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:37.616033    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mbktf" event={"ID":"40ed9426-93f6-4d39-af17-0aff9cbfc8b7","Type":"ContainerStarted","Data":"eaa851b7348e04d1c063f7254ca056453fb48b1c4c74e3c9772307733c4424a2"}
Apr 22 20:00:37.616069 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:37.616070    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mbktf" event={"ID":"40ed9426-93f6-4d39-af17-0aff9cbfc8b7","Type":"ContainerStarted","Data":"d41a1737b22eb8a8a2fa7ae0d1a7fd97c07d384582527e732c49146b34ef067d"}
Apr 22 20:00:37.634999 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:37.634955    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mbktf" podStartSLOduration=3.78691714 podStartE2EDuration="4.634942434s" podCreationTimestamp="2026-04-22 20:00:33 +0000 UTC" firstStartedPulling="2026-04-22 20:00:34.715257701 +0000 UTC m=+186.132921334" lastFinishedPulling="2026-04-22 20:00:35.563283003 +0000 UTC m=+186.980946628" observedRunningTime="2026-04-22 20:00:37.633266678 +0000 UTC m=+189.050930347" watchObservedRunningTime="2026-04-22 20:00:37.634942434 +0000 UTC m=+189.052606080"
Apr 22 20:00:39.982881 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:39.982838    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 22 20:00:39.986385 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:39.986368    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:39.988945 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:39.988922    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Apr 22 20:00:39.989078 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:39.989009    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Apr 22 20:00:39.989078 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:39.989028    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Apr 22 20:00:39.989322 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:39.989139    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Apr 22 20:00:39.989322 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:39.989290    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Apr 22 20:00:39.989402 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:39.989373    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Apr 22 20:00:39.989526 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:39.989504    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Apr 22 20:00:39.989648 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:39.989561    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Apr 22 20:00:39.990548 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:39.990525    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Apr 22 20:00:39.990665 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:39.990561    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Apr 22 20:00:39.990665 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:39.990576    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Apr 22 20:00:39.990665 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:39.990614    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Apr 22 20:00:39.990860 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:39.990770    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4391lvcbi9v7a\""
Apr 22 20:00:39.991389 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:39.991365    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7bkc2\""
Apr 22 20:00:39.992881 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:39.992857    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Apr 22 20:00:39.993928 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:39.993908    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 22 20:00:40.108172 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.108143    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.108172 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.108174    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe534fb4-52d3-46f2-9973-fcade6896ffc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.108314 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.108193    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.108314 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.108221    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.108314 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.108286    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.108457 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.108325    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.108457 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.108361    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.108457 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.108386    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.108579 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.108474    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.108579 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.108498    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.108579 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.108516    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.108579 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.108533    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pl6m\" (UniqueName: \"kubernetes.io/projected/fe534fb4-52d3-46f2-9973-fcade6896ffc-kube-api-access-4pl6m\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.108579 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.108549    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe534fb4-52d3-46f2-9973-fcade6896ffc-config-out\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.108579 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.108567    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe534fb4-52d3-46f2-9973-fcade6896ffc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.108775 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.108625    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-web-config\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.108775 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.108642    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-config\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.108775 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.108660    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.108775 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.108674    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.209859 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.209834    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-web-config\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.209936 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.209871    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-config\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.209936 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.209899    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.209936 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.209924    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.210096 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.209957    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.210096 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.209987    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe534fb4-52d3-46f2-9973-fcade6896ffc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.210096 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.210010    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.210096 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.210040    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.210096 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.210084    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.210323 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.210111    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.210323 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.210143    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.210323 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.210169    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.210323 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.210253    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.210323 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.210286    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.210592 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.210320    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.210592 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.210351    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4pl6m\" (UniqueName: \"kubernetes.io/projected/fe534fb4-52d3-46f2-9973-fcade6896ffc-kube-api-access-4pl6m\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.210592 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.210378    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe534fb4-52d3-46f2-9973-fcade6896ffc-config-out\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.210592 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.210404    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe534fb4-52d3-46f2-9973-fcade6896ffc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.210827 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.210806    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe534fb4-52d3-46f2-9973-fcade6896ffc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.211283 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.211261    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.211380 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.211360    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.213811 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.213776    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.214267 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.214240    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.214374 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.214343    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-config\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.214483 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.214382    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-web-config\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.214708 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.214677    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe534fb4-52d3-46f2-9973-fcade6896ffc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.214796 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.214728    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.215235 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.215193    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.215392 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.215369    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe534fb4-52d3-46f2-9973-fcade6896ffc-config-out\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.215757 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.215736    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.216104 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.216080    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.216207 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.216185    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.216397 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.216379    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.216505 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.216453    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.217045 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.217026    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.218769 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.218745    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pl6m\" (UniqueName: \"kubernetes.io/projected/fe534fb4-52d3-46f2-9973-fcade6896ffc-kube-api-access-4pl6m\") pod \"prometheus-k8s-0\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.296679 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.296623    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:40.419702 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.419604    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 22 20:00:40.422686 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:00:40.422657    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe534fb4_52d3_46f2_9973_fcade6896ffc.slice/crio-543b16f30aa879d506f90ee7e8c1b9167131a0d3df98eb65c17d227709083cdb WatchSource:0}: Error finding container 543b16f30aa879d506f90ee7e8c1b9167131a0d3df98eb65c17d227709083cdb: Status 404 returned error can't find the container with id 543b16f30aa879d506f90ee7e8c1b9167131a0d3df98eb65c17d227709083cdb
Apr 22 20:00:40.625187 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:40.625113    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe534fb4-52d3-46f2-9973-fcade6896ffc","Type":"ContainerStarted","Data":"543b16f30aa879d506f90ee7e8c1b9167131a0d3df98eb65c17d227709083cdb"}
Apr 22 20:00:41.629757 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:41.629733    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe534fb4-52d3-46f2-9973-fcade6896ffc","Type":"ContainerStarted","Data":"625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3"}
Apr 22 20:00:42.633659 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:42.633618    2574 generic.go:358] "Generic (PLEG): container finished" podID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerID="625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3" exitCode=0
Apr 22 20:00:42.634015 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:42.633662    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe534fb4-52d3-46f2-9973-fcade6896ffc","Type":"ContainerDied","Data":"625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3"}
Apr 22 20:00:45.641994 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:45.641913    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe534fb4-52d3-46f2-9973-fcade6896ffc","Type":"ContainerStarted","Data":"30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af"}
Apr 22 20:00:45.641994 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:45.641948    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe534fb4-52d3-46f2-9973-fcade6896ffc","Type":"ContainerStarted","Data":"8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca"}
Apr 22 20:00:47.649560 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:47.649478    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe534fb4-52d3-46f2-9973-fcade6896ffc","Type":"ContainerStarted","Data":"8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f"}
Apr 22 20:00:47.649560 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:47.649517    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe534fb4-52d3-46f2-9973-fcade6896ffc","Type":"ContainerStarted","Data":"73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b"}
Apr 22 20:00:47.649560 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:47.649532    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe534fb4-52d3-46f2-9973-fcade6896ffc","Type":"ContainerStarted","Data":"ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0"}
Apr 22 20:00:47.649560 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:47.649544    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe534fb4-52d3-46f2-9973-fcade6896ffc","Type":"ContainerStarted","Data":"25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686"}
Apr 22 20:00:47.692319 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:47.692271    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.8450523890000001 podStartE2EDuration="8.692257497s" podCreationTimestamp="2026-04-22 20:00:39 +0000 UTC" firstStartedPulling="2026-04-22 20:00:40.424556741 +0000 UTC m=+191.842220365" lastFinishedPulling="2026-04-22 20:00:47.271761849 +0000 UTC m=+198.689425473" observedRunningTime="2026-04-22 20:00:47.689799618 +0000 UTC m=+199.107463285" watchObservedRunningTime="2026-04-22 20:00:47.692257497 +0000 UTC m=+199.109921142"
Apr 22 20:00:48.474947 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.474912    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59cb695579-cvq2f"]
Apr 22 20:00:48.475112 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:00:48.475091    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-59cb695579-cvq2f" podUID="819fb49a-5bbf-4c97-ad55-ca03015622ca"
Apr 22 20:00:48.652561 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.652528    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 20:00:48.656751 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.656724    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 20:00:48.784656 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.784584    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/819fb49a-5bbf-4c97-ad55-ca03015622ca-installation-pull-secrets\") pod \"819fb49a-5bbf-4c97-ad55-ca03015622ca\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") "
Apr 22 20:00:48.784802 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.784701    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-bound-sa-token\") pod \"819fb49a-5bbf-4c97-ad55-ca03015622ca\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") "
Apr 22 20:00:48.784802 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.784750    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/819fb49a-5bbf-4c97-ad55-ca03015622ca-image-registry-private-configuration\") pod \"819fb49a-5bbf-4c97-ad55-ca03015622ca\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") "
Apr 22 20:00:48.784916 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.784859    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/819fb49a-5bbf-4c97-ad55-ca03015622ca-ca-trust-extracted\") pod \"819fb49a-5bbf-4c97-ad55-ca03015622ca\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") "
Apr 22 20:00:48.784916 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.784905    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-certificates\") pod \"819fb49a-5bbf-4c97-ad55-ca03015622ca\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") "
Apr 22 20:00:48.785022 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.784932    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffwct\" (UniqueName: \"kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-kube-api-access-ffwct\") pod \"819fb49a-5bbf-4c97-ad55-ca03015622ca\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") "
Apr 22 20:00:48.785022 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.785005    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/819fb49a-5bbf-4c97-ad55-ca03015622ca-trusted-ca\") pod \"819fb49a-5bbf-4c97-ad55-ca03015622ca\" (UID: \"819fb49a-5bbf-4c97-ad55-ca03015622ca\") "
Apr 22 20:00:48.785133 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.785098    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819fb49a-5bbf-4c97-ad55-ca03015622ca-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "819fb49a-5bbf-4c97-ad55-ca03015622ca" (UID: "819fb49a-5bbf-4c97-ad55-ca03015622ca"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:00:48.786718 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.785660    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/819fb49a-5bbf-4c97-ad55-ca03015622ca-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "819fb49a-5bbf-4c97-ad55-ca03015622ca" (UID: "819fb49a-5bbf-4c97-ad55-ca03015622ca"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:00:48.786718 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.785725    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/819fb49a-5bbf-4c97-ad55-ca03015622ca-trusted-ca\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:00:48.786718 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.785748    2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/819fb49a-5bbf-4c97-ad55-ca03015622ca-ca-trust-extracted\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:00:48.787312 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.787285    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "819fb49a-5bbf-4c97-ad55-ca03015622ca" (UID: "819fb49a-5bbf-4c97-ad55-ca03015622ca"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:00:48.787545 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.787525    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/819fb49a-5bbf-4c97-ad55-ca03015622ca-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "819fb49a-5bbf-4c97-ad55-ca03015622ca" (UID: "819fb49a-5bbf-4c97-ad55-ca03015622ca"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:00:48.787936 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.787905    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/819fb49a-5bbf-4c97-ad55-ca03015622ca-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "819fb49a-5bbf-4c97-ad55-ca03015622ca" (UID: "819fb49a-5bbf-4c97-ad55-ca03015622ca"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:00:48.787936 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.787909    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-kube-api-access-ffwct" (OuterVolumeSpecName: "kube-api-access-ffwct") pod "819fb49a-5bbf-4c97-ad55-ca03015622ca" (UID: "819fb49a-5bbf-4c97-ad55-ca03015622ca"). InnerVolumeSpecName "kube-api-access-ffwct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:00:48.788053 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.787944    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "819fb49a-5bbf-4c97-ad55-ca03015622ca" (UID: "819fb49a-5bbf-4c97-ad55-ca03015622ca"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:00:48.886593 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.886553    2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-bound-sa-token\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:00:48.886593 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.886591    2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/819fb49a-5bbf-4c97-ad55-ca03015622ca-image-registry-private-configuration\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:00:48.886593 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.886602    2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-certificates\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:00:48.886796 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.886612    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffwct\" (UniqueName: \"kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-kube-api-access-ffwct\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:00:48.886796 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:48.886622    2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/819fb49a-5bbf-4c97-ad55-ca03015622ca-installation-pull-secrets\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:00:49.655140 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:49.655110    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59cb695579-cvq2f"
Apr 22 20:00:49.686307 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:49.686276    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59cb695579-cvq2f"]
Apr 22 20:00:49.689132 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:49.689105    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59cb695579-cvq2f"]
Apr 22 20:00:49.793160 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:49.793123    2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/819fb49a-5bbf-4c97-ad55-ca03015622ca-registry-tls\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:00:50.297529 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:50.297497    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:00:51.159534 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:00:51.159499    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="819fb49a-5bbf-4c97-ad55-ca03015622ca" path="/var/lib/kubelet/pods/819fb49a-5bbf-4c97-ad55-ca03015622ca/volumes"
Apr 22 20:01:17.687503 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:17.687472    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-69db897b98-s4j8t_0aefc85e-890b-4111-b151-cdea5415289d/kube-state-metrics/0.log"
Apr 22 20:01:17.888973 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:17.888939    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-69db897b98-s4j8t_0aefc85e-890b-4111-b151-cdea5415289d/kube-rbac-proxy-main/0.log"
Apr 22 20:01:18.087476 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:18.087449    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-69db897b98-s4j8t_0aefc85e-890b-4111-b151-cdea5415289d/kube-rbac-proxy-self/0.log"
Apr 22 20:01:19.887152 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:19.887121    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mbktf_40ed9426-93f6-4d39-af17-0aff9cbfc8b7/init-textfile/0.log"
Apr 22 20:01:20.088158 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:20.088128    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mbktf_40ed9426-93f6-4d39-af17-0aff9cbfc8b7/node-exporter/0.log"
Apr 22 20:01:20.287397 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:20.287365    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mbktf_40ed9426-93f6-4d39-af17-0aff9cbfc8b7/kube-rbac-proxy/0.log"
Apr 22 20:01:21.087468 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:21.087408    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe534fb4-52d3-46f2-9973-fcade6896ffc/init-config-reloader/0.log"
Apr 22 20:01:21.288268 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:21.288241    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe534fb4-52d3-46f2-9973-fcade6896ffc/prometheus/0.log"
Apr 22 20:01:21.487823 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:21.487794    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe534fb4-52d3-46f2-9973-fcade6896ffc/config-reloader/0.log"
Apr 22 20:01:21.689163 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:21.689133    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe534fb4-52d3-46f2-9973-fcade6896ffc/thanos-sidecar/0.log"
Apr 22 20:01:21.887466 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:21.887366    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe534fb4-52d3-46f2-9973-fcade6896ffc/kube-rbac-proxy-web/0.log"
Apr 22 20:01:22.091549 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:22.091521    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe534fb4-52d3-46f2-9973-fcade6896ffc/kube-rbac-proxy/0.log"
Apr 22 20:01:22.287547 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:22.287520    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe534fb4-52d3-46f2-9973-fcade6896ffc/kube-rbac-proxy-thanos/0.log"
Apr 22 20:01:22.887022 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:22.886992    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-57cf98b594-rcmm5_88ede3f5-5ebd-4226-94d9-6a2b8687ebe6/prometheus-operator-admission-webhook/0.log"
Apr 22 20:01:26.888299 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:26.888269    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sz29m_c4edbe2f-e978-4101-8d32-e35e7e5e740e/dns-node-resolver/0.log"
Apr 22 20:01:40.296932 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:40.296900    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:40.315955 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:40.315930    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:40.814442 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:40.814404    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:40.876869 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:40.876838    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs\") pod \"network-metrics-daemon-9cnw2\" (UID: \"cdf7849a-206d-4b5a-ad81-e723c58f4426\") " pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 20:01:40.879271 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:40.879248    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdf7849a-206d-4b5a-ad81-e723c58f4426-metrics-certs\") pod \"network-metrics-daemon-9cnw2\" (UID: \"cdf7849a-206d-4b5a-ad81-e723c58f4426\") " pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 20:01:41.061244 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:41.061219    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-79t2x\""
Apr 22 20:01:41.069465 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:41.069407    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cnw2"
Apr 22 20:01:41.181260 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:41.181231    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9cnw2"]
Apr 22 20:01:41.185360 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:01:41.185330    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf7849a_206d_4b5a_ad81_e723c58f4426.slice/crio-aa10b3645f1bce2cfdc60908dc5332a98e9f20eede4288feae9b45efc1d71eb8 WatchSource:0}: Error finding container aa10b3645f1bce2cfdc60908dc5332a98e9f20eede4288feae9b45efc1d71eb8: Status 404 returned error can't find the container with id aa10b3645f1bce2cfdc60908dc5332a98e9f20eede4288feae9b45efc1d71eb8
Apr 22 20:01:41.800113 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:41.800074    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9cnw2" event={"ID":"cdf7849a-206d-4b5a-ad81-e723c58f4426","Type":"ContainerStarted","Data":"aa10b3645f1bce2cfdc60908dc5332a98e9f20eede4288feae9b45efc1d71eb8"}
Apr 22 20:01:42.804710 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:42.804677    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9cnw2" event={"ID":"cdf7849a-206d-4b5a-ad81-e723c58f4426","Type":"ContainerStarted","Data":"522cf7358381e6fcc3436309d1d2f2c4ab3ec3b14f3b5e77502c2c6645163672"}
Apr 22 20:01:42.805068 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:42.804716    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9cnw2" event={"ID":"cdf7849a-206d-4b5a-ad81-e723c58f4426","Type":"ContainerStarted","Data":"0543047ff28fa3770a4839d069841358b2d015f5c1c1a02168e0afb2d40c2489"}
Apr 22 20:01:42.820852 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:42.820800    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9cnw2" podStartSLOduration=252.905205506 podStartE2EDuration="4m13.820785468s" podCreationTimestamp="2026-04-22 19:57:29 +0000 UTC" firstStartedPulling="2026-04-22 20:01:41.187063171 +0000 UTC m=+252.604726795" lastFinishedPulling="2026-04-22 20:01:42.102643132 +0000 UTC m=+253.520306757" observedRunningTime="2026-04-22 20:01:42.81879583 +0000 UTC m=+254.236459477" watchObservedRunningTime="2026-04-22 20:01:42.820785468 +0000 UTC m=+254.238449151"
Apr 22 20:01:58.282021 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.281937    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 22 20:01:58.282569 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.282462    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="thanos-sidecar" containerID="cri-o://25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686" gracePeriod=600
Apr 22 20:01:58.282569 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.282483    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="kube-rbac-proxy" containerID="cri-o://73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b" gracePeriod=600
Apr 22 20:01:58.282569 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.282509    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="kube-rbac-proxy-web" containerID="cri-o://ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0" gracePeriod=600
Apr 22 20:01:58.282826 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.282462    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="prometheus" containerID="cri-o://8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca" gracePeriod=600
Apr 22 20:01:58.282826 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.282519    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="config-reloader" containerID="cri-o://30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af" gracePeriod=600
Apr 22 20:01:58.282826 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.282509    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="kube-rbac-proxy-thanos" containerID="cri-o://8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f" gracePeriod=600
Apr 22 20:01:58.513592 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.513571    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:58.613635 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.613552    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-configmap-kubelet-serving-ca-bundle\") pod \"fe534fb4-52d3-46f2-9973-fcade6896ffc\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") "
Apr 22 20:01:58.613635 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.613598    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-configmap-metrics-client-ca\") pod \"fe534fb4-52d3-46f2-9973-fcade6896ffc\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") "
Apr 22 20:01:58.613843 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.613637    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fe534fb4-52d3-46f2-9973-fcade6896ffc\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") "
Apr 22 20:01:58.613843 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.613750    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-metrics-client-certs\") pod \"fe534fb4-52d3-46f2-9973-fcade6896ffc\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") "
Apr 22 20:01:58.613843 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.613792    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe534fb4-52d3-46f2-9973-fcade6896ffc-tls-assets\") pod \"fe534fb4-52d3-46f2-9973-fcade6896ffc\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") "
Apr 22 20:01:58.613843 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.613829    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-kube-rbac-proxy\") pod \"fe534fb4-52d3-46f2-9973-fcade6896ffc\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") "
Apr 22 20:01:58.614035 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.613856    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fe534fb4-52d3-46f2-9973-fcade6896ffc\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") "
Apr 22 20:01:58.614035 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.613884    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-prometheus-trusted-ca-bundle\") pod \"fe534fb4-52d3-46f2-9973-fcade6896ffc\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") "
Apr 22 20:01:58.614035 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.613907    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-grpc-tls\") pod \"fe534fb4-52d3-46f2-9973-fcade6896ffc\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") "
Apr 22 20:01:58.614035 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.613932    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-web-config\") pod \"fe534fb4-52d3-46f2-9973-fcade6896ffc\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") "
Apr 22 20:01:58.614035 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.613959    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-config\") pod \"fe534fb4-52d3-46f2-9973-fcade6896ffc\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") "
Apr 22 20:01:58.614035 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.613988    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe534fb4-52d3-46f2-9973-fcade6896ffc-config-out\") pod \"fe534fb4-52d3-46f2-9973-fcade6896ffc\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") "
Apr 22 20:01:58.614035 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.614020    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-prometheus-k8s-tls\") pod \"fe534fb4-52d3-46f2-9973-fcade6896ffc\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") "
Apr 22 20:01:58.614035 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.614019    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "fe534fb4-52d3-46f2-9973-fcade6896ffc" (UID: "fe534fb4-52d3-46f2-9973-fcade6896ffc"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:01:58.614035 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.614026    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "fe534fb4-52d3-46f2-9973-fcade6896ffc" (UID: "fe534fb4-52d3-46f2-9973-fcade6896ffc"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:01:58.614666 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.614052    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-configmap-serving-certs-ca-bundle\") pod \"fe534fb4-52d3-46f2-9973-fcade6896ffc\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") "
Apr 22 20:01:58.614666 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.614087    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-prometheus-k8s-rulefiles-0\") pod \"fe534fb4-52d3-46f2-9973-fcade6896ffc\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") "
Apr 22 20:01:58.614666 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.614117    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe534fb4-52d3-46f2-9973-fcade6896ffc-prometheus-k8s-db\") pod \"fe534fb4-52d3-46f2-9973-fcade6896ffc\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") "
Apr 22 20:01:58.614666 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.614141    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-thanos-prometheus-http-client-file\") pod \"fe534fb4-52d3-46f2-9973-fcade6896ffc\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") "
Apr 22 20:01:58.614666 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.614172    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pl6m\" (UniqueName: \"kubernetes.io/projected/fe534fb4-52d3-46f2-9973-fcade6896ffc-kube-api-access-4pl6m\") pod \"fe534fb4-52d3-46f2-9973-fcade6896ffc\" (UID: \"fe534fb4-52d3-46f2-9973-fcade6896ffc\") "
Apr 22 20:01:58.614666 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.614463    2574 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:01:58.614666 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.614466    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "fe534fb4-52d3-46f2-9973-fcade6896ffc" (UID: "fe534fb4-52d3-46f2-9973-fcade6896ffc"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:01:58.614666 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.614483    2574 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-configmap-metrics-client-ca\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:01:58.616273 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.615895    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "fe534fb4-52d3-46f2-9973-fcade6896ffc" (UID: "fe534fb4-52d3-46f2-9973-fcade6896ffc"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:01:58.616510 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.616479    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "fe534fb4-52d3-46f2-9973-fcade6896ffc" (UID: "fe534fb4-52d3-46f2-9973-fcade6896ffc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:01:58.616592 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.616577    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe534fb4-52d3-46f2-9973-fcade6896ffc-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "fe534fb4-52d3-46f2-9973-fcade6896ffc" (UID: "fe534fb4-52d3-46f2-9973-fcade6896ffc"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:01:58.616757 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.616703    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "fe534fb4-52d3-46f2-9973-fcade6896ffc" (UID: "fe534fb4-52d3-46f2-9973-fcade6896ffc"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:01:58.617206 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.617169    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe534fb4-52d3-46f2-9973-fcade6896ffc-kube-api-access-4pl6m" (OuterVolumeSpecName: "kube-api-access-4pl6m") pod "fe534fb4-52d3-46f2-9973-fcade6896ffc" (UID: "fe534fb4-52d3-46f2-9973-fcade6896ffc"). InnerVolumeSpecName "kube-api-access-4pl6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:01:58.617523 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.617488    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe534fb4-52d3-46f2-9973-fcade6896ffc-config-out" (OuterVolumeSpecName: "config-out") pod "fe534fb4-52d3-46f2-9973-fcade6896ffc" (UID: "fe534fb4-52d3-46f2-9973-fcade6896ffc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:01:58.618478 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.618446    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "fe534fb4-52d3-46f2-9973-fcade6896ffc" (UID: "fe534fb4-52d3-46f2-9973-fcade6896ffc"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:01:58.618694 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.618604    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "fe534fb4-52d3-46f2-9973-fcade6896ffc" (UID: "fe534fb4-52d3-46f2-9973-fcade6896ffc"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:01:58.618694 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.618658    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-config" (OuterVolumeSpecName: "config") pod "fe534fb4-52d3-46f2-9973-fcade6896ffc" (UID: "fe534fb4-52d3-46f2-9973-fcade6896ffc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:01:58.618881 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.618856    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe534fb4-52d3-46f2-9973-fcade6896ffc-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "fe534fb4-52d3-46f2-9973-fcade6896ffc" (UID: "fe534fb4-52d3-46f2-9973-fcade6896ffc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:01:58.619001 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.618973    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "fe534fb4-52d3-46f2-9973-fcade6896ffc" (UID: "fe534fb4-52d3-46f2-9973-fcade6896ffc"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:01:58.619220 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.619203    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "fe534fb4-52d3-46f2-9973-fcade6896ffc" (UID: "fe534fb4-52d3-46f2-9973-fcade6896ffc"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:01:58.619309 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.619289    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "fe534fb4-52d3-46f2-9973-fcade6896ffc" (UID: "fe534fb4-52d3-46f2-9973-fcade6896ffc"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:01:58.619880 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.619854    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "fe534fb4-52d3-46f2-9973-fcade6896ffc" (UID: "fe534fb4-52d3-46f2-9973-fcade6896ffc"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:01:58.627290 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.627268    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-web-config" (OuterVolumeSpecName: "web-config") pod "fe534fb4-52d3-46f2-9973-fcade6896ffc" (UID: "fe534fb4-52d3-46f2-9973-fcade6896ffc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:01:58.715760 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.715740    2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:01:58.715760 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.715762    2574 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-metrics-client-certs\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:01:58.715879 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.715772    2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe534fb4-52d3-46f2-9973-fcade6896ffc-tls-assets\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:01:58.715879 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.715781    2574 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-kube-rbac-proxy\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:01:58.715879 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.715791    2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:01:58.715879 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.715800    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:01:58.715879 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.715809    2574 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-grpc-tls\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:01:58.715879 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.715818    2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-web-config\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:01:58.715879 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.715826    2574 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-config\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:01:58.715879 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.715833    2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe534fb4-52d3-46f2-9973-fcade6896ffc-config-out\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:01:58.715879 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.715842    2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:01:58.715879 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.715852    2574 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:01:58.715879 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.715860    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe534fb4-52d3-46f2-9973-fcade6896ffc-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:01:58.715879 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.715869    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe534fb4-52d3-46f2-9973-fcade6896ffc-prometheus-k8s-db\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:01:58.715879 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.715877    2574 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe534fb4-52d3-46f2-9973-fcade6896ffc-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:01:58.716229 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.715886    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pl6m\" (UniqueName: \"kubernetes.io/projected/fe534fb4-52d3-46f2-9973-fcade6896ffc-kube-api-access-4pl6m\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:01:58.846437 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.846396    2574 generic.go:358] "Generic (PLEG): container finished" podID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerID="8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f" exitCode=0
Apr 22 20:01:58.846437 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.846433    2574 generic.go:358] "Generic (PLEG): container finished" podID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerID="73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b" exitCode=0
Apr 22 20:01:58.846437 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.846439    2574 generic.go:358] "Generic (PLEG): container finished" podID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerID="ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0" exitCode=0
Apr 22 20:01:58.846606 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.846446    2574 generic.go:358] "Generic (PLEG): container finished" podID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerID="25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686" exitCode=0
Apr 22 20:01:58.846606 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.846453    2574 generic.go:358] "Generic (PLEG): container finished" podID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerID="30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af" exitCode=0
Apr 22 20:01:58.846606 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.846460    2574 generic.go:358] "Generic (PLEG): container finished" podID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerID="8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca" exitCode=0
Apr 22 20:01:58.846606 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.846482    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe534fb4-52d3-46f2-9973-fcade6896ffc","Type":"ContainerDied","Data":"8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f"}
Apr 22 20:01:58.846606 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.846515    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:58.846606 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.846522    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe534fb4-52d3-46f2-9973-fcade6896ffc","Type":"ContainerDied","Data":"73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b"}
Apr 22 20:01:58.846606 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.846533    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe534fb4-52d3-46f2-9973-fcade6896ffc","Type":"ContainerDied","Data":"ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0"}
Apr 22 20:01:58.846606 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.846542    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe534fb4-52d3-46f2-9973-fcade6896ffc","Type":"ContainerDied","Data":"25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686"}
Apr 22 20:01:58.846606 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.846552    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe534fb4-52d3-46f2-9973-fcade6896ffc","Type":"ContainerDied","Data":"30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af"}
Apr 22 20:01:58.846606 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.846562    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe534fb4-52d3-46f2-9973-fcade6896ffc","Type":"ContainerDied","Data":"8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca"}
Apr 22 20:01:58.846606 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.846574    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe534fb4-52d3-46f2-9973-fcade6896ffc","Type":"ContainerDied","Data":"543b16f30aa879d506f90ee7e8c1b9167131a0d3df98eb65c17d227709083cdb"}
Apr 22 20:01:58.846606 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.846589    2574 scope.go:117] "RemoveContainer" containerID="8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f"
Apr 22 20:01:58.854300 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.854263    2574 scope.go:117] "RemoveContainer" containerID="73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b"
Apr 22 20:01:58.860815 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.860801    2574 scope.go:117] "RemoveContainer" containerID="ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0"
Apr 22 20:01:58.866550 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.866534    2574 scope.go:117] "RemoveContainer" containerID="25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686"
Apr 22 20:01:58.869734 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.869712    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 22 20:01:58.873098 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.873076    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 22 20:01:58.873868 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.873856    2574 scope.go:117] "RemoveContainer" containerID="30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af"
Apr 22 20:01:58.879911 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.879880    2574 scope.go:117] "RemoveContainer" containerID="8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca"
Apr 22 20:01:58.886396 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.886365    2574 scope.go:117] "RemoveContainer" containerID="625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3"
Apr 22 20:01:58.892277 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.892261    2574 scope.go:117] "RemoveContainer" containerID="8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f"
Apr 22 20:01:58.892596 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:01:58.892577    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f\": container with ID starting with 8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f not found: ID does not exist" containerID="8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f"
Apr 22 20:01:58.892645 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.892605    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f"} err="failed to get container status \"8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f\": rpc error: code = NotFound desc = could not find container \"8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f\": container with ID starting with 8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f not found: ID does not exist"
Apr 22 20:01:58.892645 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.892635    2574 scope.go:117] "RemoveContainer" containerID="73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b"
Apr 22 20:01:58.892850 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:01:58.892836    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b\": container with ID starting with 73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b not found: ID does not exist" containerID="73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b"
Apr 22 20:01:58.892886 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.892853    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b"} err="failed to get container status \"73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b\": rpc error: code = NotFound desc = could not find container \"73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b\": container with ID starting with 73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b not found: ID does not exist"
Apr 22 20:01:58.892886 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.892865    2574 scope.go:117] "RemoveContainer" containerID="ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0"
Apr 22 20:01:58.893060 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:01:58.893046    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0\": container with ID starting with ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0 not found: ID does not exist" containerID="ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0"
Apr 22 20:01:58.893105 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.893063    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0"} err="failed to get container status \"ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0\": rpc error: code = NotFound desc = could not find container \"ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0\": container with ID starting with ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0 not found: ID does not exist"
Apr 22 20:01:58.893105 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.893074    2574 scope.go:117] "RemoveContainer" containerID="25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686"
Apr 22 20:01:58.893320 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:01:58.893299    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686\": container with ID starting with 25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686 not found: ID does not exist" containerID="25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686"
Apr 22 20:01:58.893392 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.893324    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686"} err="failed to get container status \"25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686\": rpc error: code = NotFound desc = could not find container \"25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686\": container with ID starting with 25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686 not found: ID does not exist"
Apr 22 20:01:58.893392 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.893344    2574 scope.go:117] "RemoveContainer" containerID="30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af"
Apr 22 20:01:58.894066 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:01:58.893743    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af\": container with ID starting with 30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af not found: ID does not exist" containerID="30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af"
Apr 22 20:01:58.894066 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.893774    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af"} err="failed to get container status \"30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af\": rpc error: code = NotFound desc = could not find container \"30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af\": container with ID starting with 30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af not found: ID does not exist"
Apr 22 20:01:58.894066 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.893798    2574 scope.go:117] "RemoveContainer" containerID="8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca"
Apr 22 20:01:58.894244 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:01:58.894146    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca\": container with ID starting with 8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca not found: ID does not exist" containerID="8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca"
Apr 22 20:01:58.894244 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.894171    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca"} err="failed to get container status \"8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca\": rpc error: code = NotFound desc = could not find container \"8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca\": container with ID starting with 8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca not found: ID does not exist"
Apr 22 20:01:58.894244 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.894186    2574 scope.go:117] "RemoveContainer" containerID="625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3"
Apr 22 20:01:58.894476 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:01:58.894454    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3\": container with ID starting with 625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3 not found: ID does not exist" containerID="625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3"
Apr 22 20:01:58.894525 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.894483    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3"} err="failed to get container status \"625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3\": rpc error: code = NotFound desc = could not find container \"625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3\": container with ID starting with 625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3 not found: ID does not exist"
Apr 22 20:01:58.894525 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.894506    2574 scope.go:117] "RemoveContainer" containerID="8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f"
Apr 22 20:01:58.894774 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.894756    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f"} err="failed to get container status \"8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f\": rpc error: code = NotFound desc = could not find container \"8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f\": container with ID starting with 8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f not found: ID does not exist"
Apr 22 20:01:58.894774 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.894774    2574 scope.go:117] "RemoveContainer" containerID="73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b"
Apr 22 20:01:58.895104 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895064    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b"} err="failed to get container status \"73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b\": rpc error: code = NotFound desc = could not find container \"73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b\": container with ID starting with 73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b not found: ID does not exist"
Apr 22 20:01:58.895160 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895106    2574 scope.go:117] "RemoveContainer" containerID="ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0"
Apr 22 20:01:58.895230 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895207    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 22 20:01:58.895393 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895369    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0"} err="failed to get container status \"ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0\": rpc error: code = NotFound desc = could not find container \"ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0\": container with ID starting with ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0 not found: ID does not exist"
Apr 22 20:01:58.895483 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895395    2574 scope.go:117] "RemoveContainer" containerID="25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686"
Apr 22 20:01:58.895543 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895519    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="prometheus"
Apr 22 20:01:58.895543 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895534    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="prometheus"
Apr 22 20:01:58.895646 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895546    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="thanos-sidecar"
Apr 22 20:01:58.895646 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895556    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="thanos-sidecar"
Apr 22 20:01:58.895646 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895565    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="kube-rbac-proxy"
Apr 22 20:01:58.895646 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895573    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="kube-rbac-proxy"
Apr 22 20:01:58.895646 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895588    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="kube-rbac-proxy-web"
Apr 22 20:01:58.895646 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895598    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="kube-rbac-proxy-web"
Apr 22 20:01:58.895646 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895611    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="init-config-reloader"
Apr 22 20:01:58.895646 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895616    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="init-config-reloader"
Apr 22 20:01:58.895646 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895625    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="config-reloader"
Apr 22 20:01:58.895646 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895633    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="config-reloader"
Apr 22 20:01:58.895646 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895644    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="kube-rbac-proxy-thanos"
Apr 22 20:01:58.895646 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895649    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="kube-rbac-proxy-thanos"
Apr 22 20:01:58.896033 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895684    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686"} err="failed to get container status \"25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686\": rpc error: code = NotFound desc = could not find container \"25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686\": container with ID starting with 25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686 not found: ID does not exist"
Apr 22 20:01:58.896033 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895702    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="kube-rbac-proxy-thanos"
Apr 22 20:01:58.896033 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895706    2574 scope.go:117] "RemoveContainer" containerID="30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af"
Apr 22 20:01:58.896033 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895716    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="kube-rbac-proxy"
Apr 22 20:01:58.896033 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895726    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="prometheus"
Apr 22 20:01:58.896033 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895733    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="thanos-sidecar"
Apr 22 20:01:58.896033 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895738    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="kube-rbac-proxy-web"
Apr 22 20:01:58.896033 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895745    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" containerName="config-reloader"
Apr 22 20:01:58.896033 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895944    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af"} err="failed to get container status \"30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af\": rpc error: code = NotFound desc = could not find container \"30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af\": container with ID starting with 30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af not found: ID does not exist"
Apr 22 20:01:58.896033 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.895967    2574 scope.go:117] "RemoveContainer" containerID="8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca"
Apr 22 20:01:58.896340 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.896178    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca"} err="failed to get container status \"8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca\": rpc error: code = NotFound desc = could not find container \"8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca\": container with ID starting with 8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca not found: ID does not exist"
Apr 22 20:01:58.896340 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.896196    2574 scope.go:117] "RemoveContainer" containerID="625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3"
Apr 22 20:01:58.896443 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.896397    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3"} err="failed to get container status \"625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3\": rpc error: code = NotFound desc = could not find container \"625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3\": container with ID starting with 625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3 not found: ID does not exist"
Apr 22 20:01:58.896443 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.896436    2574 scope.go:117] "RemoveContainer" containerID="8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f"
Apr 22 20:01:58.896642 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.896623    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f"} err="failed to get container status \"8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f\": rpc error: code = NotFound desc = could not find container \"8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f\": container with ID starting with 8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f not found: ID does not exist"
Apr 22 20:01:58.896712 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.896645    2574 scope.go:117] "RemoveContainer" containerID="73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b"
Apr 22 20:01:58.896855 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.896838    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b"} err="failed to get container status \"73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b\": rpc error: code = NotFound desc = could not find container \"73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b\": container with ID starting with 73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b not found: ID does not exist"
Apr 22 20:01:58.896917 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.896856    2574 scope.go:117] "RemoveContainer" containerID="ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0"
Apr 22 20:01:58.897062 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.897045    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0"} err="failed to get container status \"ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0\": rpc error: code = NotFound desc = could not find container \"ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0\": container with ID starting with ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0 not found: ID does not exist"
Apr 22 20:01:58.897109 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.897063    2574 scope.go:117] "RemoveContainer" containerID="25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686"
Apr 22 20:01:58.897233 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.897214    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686"} err="failed to get container status \"25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686\": rpc error: code = NotFound desc = could not find container \"25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686\": container with ID starting with 25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686 not found: ID does not exist"
Apr 22 20:01:58.897273 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.897233    2574 scope.go:117] "RemoveContainer" containerID="30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af"
Apr 22 20:01:58.897446 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.897425    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af"} err="failed to get container status \"30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af\": rpc error: code = NotFound desc = could not find container \"30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af\": container with ID starting with 30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af not found: ID does not exist"
Apr 22 20:01:58.897584 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.897449    2574 scope.go:117] "RemoveContainer" containerID="8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca"
Apr 22 20:01:58.897654 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.897631    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca"} err="failed to get container status \"8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca\": rpc error: code = NotFound desc = could not find container \"8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca\": container with ID starting with 8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca not found: ID does not exist"
Apr 22 20:01:58.897695 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.897655    2574 scope.go:117] "RemoveContainer" containerID="625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3"
Apr 22 20:01:58.897841 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.897825    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3"} err="failed to get container status \"625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3\": rpc error: code = NotFound desc = could not find container \"625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3\": container with ID starting with 625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3 not found: ID does not exist"
Apr 22 20:01:58.897841 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.897840    2574 scope.go:117] "RemoveContainer" containerID="8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f"
Apr 22 20:01:58.898008 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.897993    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f"} err="failed to get container status \"8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f\": rpc error: code = NotFound desc = could not find container \"8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f\": container with ID starting with 8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f not found: ID does not exist"
Apr 22 20:01:58.898045 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.898008    2574 scope.go:117] "RemoveContainer" containerID="73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b"
Apr 22 20:01:58.898198 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.898181    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b"} err="failed to get container status \"73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b\": rpc error: code = NotFound desc = could not find container \"73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b\": container with ID starting with 73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b not found: ID does not exist"
Apr 22 20:01:58.898242 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.898199    2574 scope.go:117] "RemoveContainer" containerID="ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0"
Apr 22 20:01:58.898370 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.898351    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0"} err="failed to get container status \"ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0\": rpc error: code = NotFound desc = could not find container \"ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0\": container with ID starting with ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0 not found: ID does not exist"
Apr 22 20:01:58.898442 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.898370    2574 scope.go:117] "RemoveContainer" containerID="25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686"
Apr 22 20:01:58.898550 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.898530    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686"} err="failed to get container status \"25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686\": rpc error: code = NotFound desc = could not find container \"25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686\": container with ID starting with 25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686 not found: ID does not exist"
Apr 22 20:01:58.898591 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.898551    2574 scope.go:117] "RemoveContainer" containerID="30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af"
Apr 22 20:01:58.898772 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.898756    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af"} err="failed to get container status \"30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af\": rpc error: code = NotFound desc = could not find container \"30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af\": container with ID starting with 30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af not found: ID does not exist"
Apr 22 20:01:58.898812 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.898773    2574 scope.go:117] "RemoveContainer" containerID="8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca"
Apr 22 20:01:58.898993 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.898976    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca"} err="failed to get container status \"8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca\": rpc error: code = NotFound desc = could not find container \"8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca\": container with ID starting with 8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca not found: ID does not exist"
Apr 22 20:01:58.899036 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.898993    2574 scope.go:117] "RemoveContainer" containerID="625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3"
Apr 22 20:01:58.899180 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.899155    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3"} err="failed to get container status \"625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3\": rpc error: code = NotFound desc = could not find container \"625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3\": container with ID starting with 625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3 not found: ID does not exist"
Apr 22 20:01:58.899180 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.899171    2574 scope.go:117] "RemoveContainer" containerID="8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f"
Apr 22 20:01:58.899373 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.899356    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f"} err="failed to get container status \"8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f\": rpc error: code = NotFound desc = could not find container \"8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f\": container with ID starting with 8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f not found: ID does not exist"
Apr 22 20:01:58.899373 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.899373    2574 scope.go:117] "RemoveContainer" containerID="73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b"
Apr 22 20:01:58.899622 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.899603    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b"} err="failed to get container status \"73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b\": rpc error: code = NotFound desc = could not find container \"73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b\": container with ID starting with 73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b not found: ID does not exist"
Apr 22 20:01:58.899684 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.899633    2574 scope.go:117] "RemoveContainer" containerID="ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0"
Apr 22 20:01:58.899813 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.899796    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:58.899874 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.899859    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0"} err="failed to get container status \"ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0\": rpc error: code = NotFound desc = could not find container \"ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0\": container with ID starting with ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0 not found: ID does not exist"
Apr 22 20:01:58.899920 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.899875    2574 scope.go:117] "RemoveContainer" containerID="25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686"
Apr 22 20:01:58.900080 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.900057    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686"} err="failed to get container status \"25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686\": rpc error: code = NotFound desc = could not find container \"25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686\": container with ID starting with 25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686 not found: ID does not exist"
Apr 22 20:01:58.900145 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.900082    2574 scope.go:117] "RemoveContainer" containerID="30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af"
Apr 22 20:01:58.900342 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.900289    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af"} err="failed to get container status \"30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af\": rpc error: code = NotFound desc = could not find container \"30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af\": container with ID starting with 30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af not found: ID does not exist"
Apr 22 20:01:58.900342 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.900341    2574 scope.go:117] "RemoveContainer" containerID="8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca"
Apr 22 20:01:58.900615 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.900597    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca"} err="failed to get container status \"8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca\": rpc error: code = NotFound desc = could not find container \"8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca\": container with ID starting with 8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca not found: ID does not exist"
Apr 22 20:01:58.900681 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.900616    2574 scope.go:117] "RemoveContainer" containerID="625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3"
Apr 22 20:01:58.900852 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.900832    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3"} err="failed to get container status \"625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3\": rpc error: code = NotFound desc = could not find container \"625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3\": container with ID starting with 625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3 not found: ID does not exist"
Apr 22 20:01:58.900897 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.900854    2574 scope.go:117] "RemoveContainer" containerID="8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f"
Apr 22 20:01:58.901054 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.901038    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f"} err="failed to get container status \"8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f\": rpc error: code = NotFound desc = could not find container \"8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f\": container with ID starting with 8cea737b22a60a3c8f5e1e56ef0cfac89b9dc72a2b1fdea5893b8c62ca0b292f not found: ID does not exist"
Apr 22 20:01:58.901099 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.901054    2574 scope.go:117] "RemoveContainer" containerID="73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b"
Apr 22 20:01:58.901275 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.901257    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b"} err="failed to get container status \"73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b\": rpc error: code = NotFound desc = could not find container \"73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b\": container with ID starting with 73f593a17f58b4b443e2a15ac922b58576acc2bbee866eb9e4ec28880203b68b not found: ID does not exist"
Apr 22 20:01:58.901313 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.901276    2574 scope.go:117] "RemoveContainer" containerID="ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0"
Apr 22 20:01:58.901526 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.901510    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0"} err="failed to get container status \"ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0\": rpc error: code = NotFound desc = could not find container \"ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0\": container with ID starting with ab5ae88be762b11e77f3683136ae1645ab4656d014faa00024752a216e19bea0 not found: ID does not exist"
Apr 22 20:01:58.901577 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.901527    2574 scope.go:117] "RemoveContainer" containerID="25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686"
Apr 22 20:01:58.901742 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.901724    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686"} err="failed to get container status \"25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686\": rpc error: code = NotFound desc = could not find container \"25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686\": container with ID starting with 25b075199a4abc327e5b2c803962ac5a8cb8fc15ce0a1e95926414346de93686 not found: ID does not exist"
Apr 22 20:01:58.901785 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.901742    2574 scope.go:117] "RemoveContainer" containerID="30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af"
Apr 22 20:01:58.901969 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.901952    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af"} err="failed to get container status \"30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af\": rpc error: code = NotFound desc = could not find container \"30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af\": container with ID starting with 30eb4e196f69c564a06dd4e920e1c5b96c8c2d5a9bc39c72f28bcf408b2d40af not found: ID does not exist"
Apr 22 20:01:58.902012 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.901969    2574 scope.go:117] "RemoveContainer" containerID="8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca"
Apr 22 20:01:58.902167 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.902152    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca"} err="failed to get container status \"8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca\": rpc error: code = NotFound desc = could not find container \"8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca\": container with ID starting with 8e76d2476aa13a34e76e672405d7f5f31474b3bd6ba9ace877bb0c8ee6539eca not found: ID does not exist"
Apr 22 20:01:58.902207 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.902167    2574 scope.go:117] "RemoveContainer" containerID="625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3"
Apr 22 20:01:58.902384 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.902367    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3"} err="failed to get container status \"625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3\": rpc error: code = NotFound desc = could not find container \"625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3\": container with ID starting with 625f97cc50a495e4f87258f5fefeb7757e133a3c2ce2aa52aba1025f78111bf3 not found: ID does not exist"
Apr 22 20:01:58.902765 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.902704    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Apr 22 20:01:58.902765 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.902718    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Apr 22 20:01:58.902765 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.902740    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Apr 22 20:01:58.902765 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.902718    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Apr 22 20:01:58.903007 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.902890    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Apr 22 20:01:58.903007 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.902975    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Apr 22 20:01:58.903629 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.903214    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Apr 22 20:01:58.903629 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.903227    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Apr 22 20:01:58.903629 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.903240    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Apr 22 20:01:58.903629 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.903250    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Apr 22 20:01:58.903629 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.903270    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Apr 22 20:01:58.903629 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.903216    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4391lvcbi9v7a\""
Apr 22 20:01:58.904014 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.903703    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7bkc2\""
Apr 22 20:01:58.906695 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.906675    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Apr 22 20:01:58.908848 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.908778    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Apr 22 20:01:58.910648 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:58.910628    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 22 20:01:59.018348 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.018315    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.018348 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.018346    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c65017f-d1d5-443b-bf4d-6129241ccc09-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.018566 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.018364    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.018566 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.018381    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.018566 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.018484    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c65017f-d1d5-443b-bf4d-6129241ccc09-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.018566 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.018532    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c65017f-d1d5-443b-bf4d-6129241ccc09-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.018566 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.018554    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.018764 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.018589    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.018764 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.018603    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.018764 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.018654    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c65017f-d1d5-443b-bf4d-6129241ccc09-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.018764 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.018705    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c65017f-d1d5-443b-bf4d-6129241ccc09-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.018764 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.018733    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-config\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.018764 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.018760    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c65017f-d1d5-443b-bf4d-6129241ccc09-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.019030 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.018801    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.019030 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.018836    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c65017f-d1d5-443b-bf4d-6129241ccc09-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.019030 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.018876    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c65017f-d1d5-443b-bf4d-6129241ccc09-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.019030 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.018910    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.019030 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.018936    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq55j\" (UniqueName: \"kubernetes.io/projected/0c65017f-d1d5-443b-bf4d-6129241ccc09-kube-api-access-nq55j\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.119618 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.119536    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c65017f-d1d5-443b-bf4d-6129241ccc09-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.119618 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.119566    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.119618 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.119594    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.119618 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.119613    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.119916 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.119636    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c65017f-d1d5-443b-bf4d-6129241ccc09-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.119916 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.119774    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c65017f-d1d5-443b-bf4d-6129241ccc09-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.119916 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.119830    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-config\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.119916 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.119876    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c65017f-d1d5-443b-bf4d-6129241ccc09-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.119916 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.119906    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.120158 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.119930    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c65017f-d1d5-443b-bf4d-6129241ccc09-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.120158 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.119961    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c65017f-d1d5-443b-bf4d-6129241ccc09-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.120158 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.119996    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.120158 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.120043    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nq55j\" (UniqueName: \"kubernetes.io/projected/0c65017f-d1d5-443b-bf4d-6129241ccc09-kube-api-access-nq55j\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.120158 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.120071    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.120158 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.120108    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c65017f-d1d5-443b-bf4d-6129241ccc09-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.120158 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.120138    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.120687 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.120164    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.120687 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.120201    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c65017f-d1d5-443b-bf4d-6129241ccc09-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.120805 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.120783    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c65017f-d1d5-443b-bf4d-6129241ccc09-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.122332 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.121214    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c65017f-d1d5-443b-bf4d-6129241ccc09-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.122494 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.122451    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c65017f-d1d5-443b-bf4d-6129241ccc09-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.122494 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.122453    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c65017f-d1d5-443b-bf4d-6129241ccc09-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.123369 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.123340    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c65017f-d1d5-443b-bf4d-6129241ccc09-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.123592 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.123569    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c65017f-d1d5-443b-bf4d-6129241ccc09-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.123864 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.123841    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c65017f-d1d5-443b-bf4d-6129241ccc09-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.123962 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.123936    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-config\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.124090 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.124064    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c65017f-d1d5-443b-bf4d-6129241ccc09-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.124502 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.124476    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.124980 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.124963    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.125062 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.125031    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.125712 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.125686    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.126216 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.126194    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.126289 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.126261    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.126590 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.126573    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.127552 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.127530    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c65017f-d1d5-443b-bf4d-6129241ccc09-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.129867 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.129849    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq55j\" (UniqueName: \"kubernetes.io/projected/0c65017f-d1d5-443b-bf4d-6129241ccc09-kube-api-access-nq55j\") pod \"prometheus-k8s-0\" (UID: \"0c65017f-d1d5-443b-bf4d-6129241ccc09\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.159060 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.159035    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe534fb4-52d3-46f2-9973-fcade6896ffc" path="/var/lib/kubelet/pods/fe534fb4-52d3-46f2-9973-fcade6896ffc/volumes"
Apr 22 20:01:59.210226 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.210205    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:01:59.334655 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.334629    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 22 20:01:59.336200 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:01:59.336166    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c65017f_d1d5_443b_bf4d_6129241ccc09.slice/crio-7673b9dea992c01136cb3b7b54cab505615d411fa577e8f58e14cdea455765e7 WatchSource:0}: Error finding container 7673b9dea992c01136cb3b7b54cab505615d411fa577e8f58e14cdea455765e7: Status 404 returned error can't find the container with id 7673b9dea992c01136cb3b7b54cab505615d411fa577e8f58e14cdea455765e7
Apr 22 20:01:59.851423 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.851373    2574 generic.go:358] "Generic (PLEG): container finished" podID="0c65017f-d1d5-443b-bf4d-6129241ccc09" containerID="ebbd41c0b4c6f49c0c2b4dcb3ccfd27c5717f52052f280e5481cdc06722372a6" exitCode=0
Apr 22 20:01:59.851560 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.851470    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c65017f-d1d5-443b-bf4d-6129241ccc09","Type":"ContainerDied","Data":"ebbd41c0b4c6f49c0c2b4dcb3ccfd27c5717f52052f280e5481cdc06722372a6"}
Apr 22 20:01:59.851560 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:01:59.851500    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c65017f-d1d5-443b-bf4d-6129241ccc09","Type":"ContainerStarted","Data":"7673b9dea992c01136cb3b7b54cab505615d411fa577e8f58e14cdea455765e7"}
Apr 22 20:02:00.857039 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:00.857007    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c65017f-d1d5-443b-bf4d-6129241ccc09","Type":"ContainerStarted","Data":"9ae30714534b3276da376e47b124324d1f342fdae747440b95a701d0426a17ad"}
Apr 22 20:02:00.857039 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:00.857043    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c65017f-d1d5-443b-bf4d-6129241ccc09","Type":"ContainerStarted","Data":"5563e4c63e98d155137cc40460c9b1dfc36f3b2e2437b5deafbb814c1427712e"}
Apr 22 20:02:00.857438 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:00.857052    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c65017f-d1d5-443b-bf4d-6129241ccc09","Type":"ContainerStarted","Data":"253f593cff7a401e6ed1bb590d4478c0073acb799a964ec62de5e67a4db848d8"}
Apr 22 20:02:00.857438 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:00.857062    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c65017f-d1d5-443b-bf4d-6129241ccc09","Type":"ContainerStarted","Data":"21d9ddc075c194d519c133554dce8405797d32d662a3412385362d8466ea8eae"}
Apr 22 20:02:00.857438 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:00.857070    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c65017f-d1d5-443b-bf4d-6129241ccc09","Type":"ContainerStarted","Data":"81c6b374b71ae5261007c1c0b5ff0b80cd12e1196b5a055de8697a7142537e42"}
Apr 22 20:02:00.857438 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:00.857080    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c65017f-d1d5-443b-bf4d-6129241ccc09","Type":"ContainerStarted","Data":"dc03bc89e2e7a5b78cf18dc7b986838ff970bb7fd52ca9d5485735efae94836f"}
Apr 22 20:02:00.885003 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:00.884928    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.884909413 podStartE2EDuration="2.884909413s" podCreationTimestamp="2026-04-22 20:01:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:02:00.883131964 +0000 UTC m=+272.300795620" watchObservedRunningTime="2026-04-22 20:02:00.884909413 +0000 UTC m=+272.302573059"
Apr 22 20:02:04.211205 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:04.211172    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:02:08.528032 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:02:08.527989    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r" podUID="173cdd12-cd9e-403c-a2b2-e18a20be58a4"
Apr 22 20:02:08.528032 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:02:08.528024    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-ct7n5" podUID="3a7698c5-bde7-4c7c-84ac-bcdba6d86851"
Apr 22 20:02:08.528597 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:02:08.527995    2574 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fs69p" podUID="054ccdff-220c-47af-b943-4797d37b8723"
Apr 22 20:02:08.880123 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:08.880048    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"
Apr 22 20:02:08.880263 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:08.880053    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fs69p"
Apr 22 20:02:08.880263 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:08.880053    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ct7n5"
Apr 22 20:02:12.015869 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:12.015832    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert\") pod \"ingress-canary-ct7n5\" (UID: \"3a7698c5-bde7-4c7c-84ac-bcdba6d86851\") " pod="openshift-ingress-canary/ingress-canary-ct7n5"
Apr 22 20:02:12.015869 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:12.015878    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls\") pod \"dns-default-fs69p\" (UID: \"054ccdff-220c-47af-b943-4797d37b8723\") " pod="openshift-dns/dns-default-fs69p"
Apr 22 20:02:12.016489 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:12.015899    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-27n8r\" (UID: \"173cdd12-cd9e-403c-a2b2-e18a20be58a4\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"
Apr 22 20:02:12.018476 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:12.018454    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/054ccdff-220c-47af-b943-4797d37b8723-metrics-tls\") pod \"dns-default-fs69p\" (UID: \"054ccdff-220c-47af-b943-4797d37b8723\") " pod="openshift-dns/dns-default-fs69p"
Apr 22 20:02:12.018577 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:12.018546    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a7698c5-bde7-4c7c-84ac-bcdba6d86851-cert\") pod \"ingress-canary-ct7n5\" (UID: \"3a7698c5-bde7-4c7c-84ac-bcdba6d86851\") " pod="openshift-ingress-canary/ingress-canary-ct7n5"
Apr 22 20:02:12.018617 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:12.018592    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/173cdd12-cd9e-403c-a2b2-e18a20be58a4-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-27n8r\" (UID: \"173cdd12-cd9e-403c-a2b2-e18a20be58a4\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"
Apr 22 20:02:12.184534 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:12.184504    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tzbrr\""
Apr 22 20:02:12.184683 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:12.184588    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lxbw7\""
Apr 22 20:02:12.184683 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:12.184588    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-864n4\""
Apr 22 20:02:12.190985 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:12.190966    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fs69p"
Apr 22 20:02:12.191063 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:12.190984    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ct7n5"
Apr 22 20:02:12.191126 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:12.190968    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"
Apr 22 20:02:12.353051 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:12.353022    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ct7n5"]
Apr 22 20:02:12.354598 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:12.354458    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-cb95c66f6-27n8r"]
Apr 22 20:02:12.368080 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:12.368060    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fs69p"]
Apr 22 20:02:12.371583 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:02:12.371551    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054ccdff_220c_47af_b943_4797d37b8723.slice/crio-9769ed4acca2751681eac87d5316a283da8dcdc4c0de1457f98766d9505e5b8b WatchSource:0}: Error finding container 9769ed4acca2751681eac87d5316a283da8dcdc4c0de1457f98766d9505e5b8b: Status 404 returned error can't find the container with id 9769ed4acca2751681eac87d5316a283da8dcdc4c0de1457f98766d9505e5b8b
Apr 22 20:02:12.890797 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:12.890761    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ct7n5" event={"ID":"3a7698c5-bde7-4c7c-84ac-bcdba6d86851","Type":"ContainerStarted","Data":"9537bd4f0f034fe5b1308f78fc4237298251184f621eb7eb4772eaa9896ec02f"}
Apr 22 20:02:12.891973 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:12.891942    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fs69p" event={"ID":"054ccdff-220c-47af-b943-4797d37b8723","Type":"ContainerStarted","Data":"9769ed4acca2751681eac87d5316a283da8dcdc4c0de1457f98766d9505e5b8b"}
Apr 22 20:02:12.893024 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:12.892989    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r" event={"ID":"173cdd12-cd9e-403c-a2b2-e18a20be58a4","Type":"ContainerStarted","Data":"1c4f4aba800255d6b7414d507191c86f07e8bb1327b5aff3b5d8f02b330c2f22"}
Apr 22 20:02:14.900063 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:14.900025    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ct7n5" event={"ID":"3a7698c5-bde7-4c7c-84ac-bcdba6d86851","Type":"ContainerStarted","Data":"f2ead51b0609140af2b7ffd0cba99a6f819863760440292ce20fab274c5e5def"}
Apr 22 20:02:14.901727 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:14.901693    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fs69p" event={"ID":"054ccdff-220c-47af-b943-4797d37b8723","Type":"ContainerStarted","Data":"dab5d9626e7934a77a2cf80479906cbaf49bd750267d33f15766fd92267c122e"}
Apr 22 20:02:14.901851 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:14.901731    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fs69p" event={"ID":"054ccdff-220c-47af-b943-4797d37b8723","Type":"ContainerStarted","Data":"e66305effb47545c566ddbebaf58fa4c5c1188b6ea8b0b4c80717ed73fb63662"}
Apr 22 20:02:14.901851 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:14.901807    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fs69p"
Apr 22 20:02:14.903097 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:14.903078    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r" event={"ID":"173cdd12-cd9e-403c-a2b2-e18a20be58a4","Type":"ContainerStarted","Data":"46fd64ab276aacd31ae35f8b35effa7cdd2c864299fbbc3f70b6d7fac13b33ca"}
Apr 22 20:02:14.915660 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:14.915611    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ct7n5" podStartSLOduration=251.529130083 podStartE2EDuration="4m13.915597336s" podCreationTimestamp="2026-04-22 19:58:01 +0000 UTC" firstStartedPulling="2026-04-22 20:02:12.353723678 +0000 UTC m=+283.771387306" lastFinishedPulling="2026-04-22 20:02:14.740190929 +0000 UTC m=+286.157854559" observedRunningTime="2026-04-22 20:02:14.915087634 +0000 UTC m=+286.332751281" watchObservedRunningTime="2026-04-22 20:02:14.915597336 +0000 UTC m=+286.333260976"
Apr 22 20:02:14.930376 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:14.930335    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-cb95c66f6-27n8r" podStartSLOduration=265.454901574 podStartE2EDuration="4m26.930321817s" podCreationTimestamp="2026-04-22 19:57:48 +0000 UTC" firstStartedPulling="2026-04-22 20:02:12.354756933 +0000 UTC m=+283.772420559" lastFinishedPulling="2026-04-22 20:02:13.830177171 +0000 UTC m=+285.247840802" observedRunningTime="2026-04-22 20:02:14.928664155 +0000 UTC m=+286.346327803" watchObservedRunningTime="2026-04-22 20:02:14.930321817 +0000 UTC m=+286.347985463"
Apr 22 20:02:14.944983 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:14.944942    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-fs69p" podStartSLOduration=252.484180215 podStartE2EDuration="4m13.944928825s" podCreationTimestamp="2026-04-22 19:58:01 +0000 UTC" firstStartedPulling="2026-04-22 20:02:12.373123489 +0000 UTC m=+283.790787113" lastFinishedPulling="2026-04-22 20:02:13.833872095 +0000 UTC m=+285.251535723" observedRunningTime="2026-04-22 20:02:14.943921355 +0000 UTC m=+286.361585103" watchObservedRunningTime="2026-04-22 20:02:14.944928825 +0000 UTC m=+286.362592470"
Apr 22 20:02:24.908018 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:24.907990    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fs69p"
Apr 22 20:02:29.034528 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:29.034506    2574 kubelet.go:1628] "Image garbage collection succeeded"
Apr 22 20:02:59.211323 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:59.211290    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:02:59.226717 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:02:59.226693    2574 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:03:00.037615 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:03:00.037587    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Apr 22 20:04:17.509779 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:17.509744    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n"]
Apr 22 20:04:17.512923 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:17.512904    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n"
Apr 22 20:04:17.515750 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:17.515728    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Apr 22 20:04:17.515848 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:17.515729    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Apr 22 20:04:17.516910 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:17.516890    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6slmt\""
Apr 22 20:04:17.520717 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:17.520698    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n"]
Apr 22 20:04:17.618027 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:17.617988    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45d1b8ae-bec1-4ba6-aeef-11b45aadf659-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n\" (UID: \"45d1b8ae-bec1-4ba6-aeef-11b45aadf659\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n"
Apr 22 20:04:17.618027 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:17.618032    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzllz\" (UniqueName: \"kubernetes.io/projected/45d1b8ae-bec1-4ba6-aeef-11b45aadf659-kube-api-access-wzllz\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n\" (UID: \"45d1b8ae-bec1-4ba6-aeef-11b45aadf659\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n"
Apr 22 20:04:17.618239 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:17.618076    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45d1b8ae-bec1-4ba6-aeef-11b45aadf659-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n\" (UID: \"45d1b8ae-bec1-4ba6-aeef-11b45aadf659\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n"
Apr 22 20:04:17.718843 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:17.718799    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45d1b8ae-bec1-4ba6-aeef-11b45aadf659-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n\" (UID: \"45d1b8ae-bec1-4ba6-aeef-11b45aadf659\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n"
Apr 22 20:04:17.718959 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:17.718855    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wzllz\" (UniqueName: \"kubernetes.io/projected/45d1b8ae-bec1-4ba6-aeef-11b45aadf659-kube-api-access-wzllz\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n\" (UID: \"45d1b8ae-bec1-4ba6-aeef-11b45aadf659\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n"
Apr 22 20:04:17.718959 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:17.718895    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45d1b8ae-bec1-4ba6-aeef-11b45aadf659-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n\" (UID: \"45d1b8ae-bec1-4ba6-aeef-11b45aadf659\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n"
Apr 22 20:04:17.719199 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:17.719179    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45d1b8ae-bec1-4ba6-aeef-11b45aadf659-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n\" (UID: \"45d1b8ae-bec1-4ba6-aeef-11b45aadf659\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n"
Apr 22 20:04:17.719236 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:17.719210    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45d1b8ae-bec1-4ba6-aeef-11b45aadf659-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n\" (UID: \"45d1b8ae-bec1-4ba6-aeef-11b45aadf659\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n"
Apr 22 20:04:17.727608 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:17.727579    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzllz\" (UniqueName: \"kubernetes.io/projected/45d1b8ae-bec1-4ba6-aeef-11b45aadf659-kube-api-access-wzllz\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n\" (UID: \"45d1b8ae-bec1-4ba6-aeef-11b45aadf659\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n"
Apr 22 20:04:17.822767 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:17.822673    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n"
Apr 22 20:04:17.938905 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:17.938785    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n"]
Apr 22 20:04:17.941592 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:04:17.941570    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d1b8ae_bec1_4ba6_aeef_11b45aadf659.slice/crio-a03f3be456714f0e1a470afccc7a013d80fed8c275c8742ac1a780729df16a0d WatchSource:0}: Error finding container a03f3be456714f0e1a470afccc7a013d80fed8c275c8742ac1a780729df16a0d: Status 404 returned error can't find the container with id a03f3be456714f0e1a470afccc7a013d80fed8c275c8742ac1a780729df16a0d
Apr 22 20:04:17.943498 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:17.943482    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 22 20:04:18.225184 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:18.225150    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n" event={"ID":"45d1b8ae-bec1-4ba6-aeef-11b45aadf659","Type":"ContainerStarted","Data":"a03f3be456714f0e1a470afccc7a013d80fed8c275c8742ac1a780729df16a0d"}
Apr 22 20:04:24.244245 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:24.244209    2574 generic.go:358] "Generic (PLEG): container finished" podID="45d1b8ae-bec1-4ba6-aeef-11b45aadf659" containerID="c861576921da5345930b827e2134a530c9f19f1bf94a35ebfd8110c2d36b77ad" exitCode=0
Apr 22 20:04:24.244585 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:24.244259    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n" event={"ID":"45d1b8ae-bec1-4ba6-aeef-11b45aadf659","Type":"ContainerDied","Data":"c861576921da5345930b827e2134a530c9f19f1bf94a35ebfd8110c2d36b77ad"}
Apr 22 20:04:27.253925 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:27.253886    2574 generic.go:358] "Generic (PLEG): container finished" podID="45d1b8ae-bec1-4ba6-aeef-11b45aadf659" containerID="778fedcb61bf63458d1041a2982546e0c90bb750ea8f21121f8f943cbfba7eb0" exitCode=0
Apr 22 20:04:27.254286 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:27.253958    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n" event={"ID":"45d1b8ae-bec1-4ba6-aeef-11b45aadf659","Type":"ContainerDied","Data":"778fedcb61bf63458d1041a2982546e0c90bb750ea8f21121f8f943cbfba7eb0"}
Apr 22 20:04:34.275717 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:34.275672    2574 generic.go:358] "Generic (PLEG): container finished" podID="45d1b8ae-bec1-4ba6-aeef-11b45aadf659" containerID="5ea38173b94b12c5652772fe7877f342809bda5260d650a9407e78467ea4b4a7" exitCode=0
Apr 22 20:04:34.276107 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:34.275740    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n" event={"ID":"45d1b8ae-bec1-4ba6-aeef-11b45aadf659","Type":"ContainerDied","Data":"5ea38173b94b12c5652772fe7877f342809bda5260d650a9407e78467ea4b4a7"}
Apr 22 20:04:35.402203 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:35.402183    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n"
Apr 22 20:04:35.570275 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:35.570189    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45d1b8ae-bec1-4ba6-aeef-11b45aadf659-bundle\") pod \"45d1b8ae-bec1-4ba6-aeef-11b45aadf659\" (UID: \"45d1b8ae-bec1-4ba6-aeef-11b45aadf659\") "
Apr 22 20:04:35.570444 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:35.570293    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45d1b8ae-bec1-4ba6-aeef-11b45aadf659-util\") pod \"45d1b8ae-bec1-4ba6-aeef-11b45aadf659\" (UID: \"45d1b8ae-bec1-4ba6-aeef-11b45aadf659\") "
Apr 22 20:04:35.570444 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:35.570351    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzllz\" (UniqueName: \"kubernetes.io/projected/45d1b8ae-bec1-4ba6-aeef-11b45aadf659-kube-api-access-wzllz\") pod \"45d1b8ae-bec1-4ba6-aeef-11b45aadf659\" (UID: \"45d1b8ae-bec1-4ba6-aeef-11b45aadf659\") "
Apr 22 20:04:35.570831 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:35.570803    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d1b8ae-bec1-4ba6-aeef-11b45aadf659-bundle" (OuterVolumeSpecName: "bundle") pod "45d1b8ae-bec1-4ba6-aeef-11b45aadf659" (UID: "45d1b8ae-bec1-4ba6-aeef-11b45aadf659"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:04:35.572729 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:35.572701    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d1b8ae-bec1-4ba6-aeef-11b45aadf659-kube-api-access-wzllz" (OuterVolumeSpecName: "kube-api-access-wzllz") pod "45d1b8ae-bec1-4ba6-aeef-11b45aadf659" (UID: "45d1b8ae-bec1-4ba6-aeef-11b45aadf659"). InnerVolumeSpecName "kube-api-access-wzllz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:04:35.574141 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:35.574111    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d1b8ae-bec1-4ba6-aeef-11b45aadf659-util" (OuterVolumeSpecName: "util") pod "45d1b8ae-bec1-4ba6-aeef-11b45aadf659" (UID: "45d1b8ae-bec1-4ba6-aeef-11b45aadf659"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:04:35.671115 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:35.671085    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzllz\" (UniqueName: \"kubernetes.io/projected/45d1b8ae-bec1-4ba6-aeef-11b45aadf659-kube-api-access-wzllz\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:04:35.671115 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:35.671109    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45d1b8ae-bec1-4ba6-aeef-11b45aadf659-bundle\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:04:35.671115 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:35.671119    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45d1b8ae-bec1-4ba6-aeef-11b45aadf659-util\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:04:36.283608 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:36.283576    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n" event={"ID":"45d1b8ae-bec1-4ba6-aeef-11b45aadf659","Type":"ContainerDied","Data":"a03f3be456714f0e1a470afccc7a013d80fed8c275c8742ac1a780729df16a0d"}
Apr 22 20:04:36.283608 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:36.283606    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a03f3be456714f0e1a470afccc7a013d80fed8c275c8742ac1a780729df16a0d"
Apr 22 20:04:36.283818 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:04:36.283613    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cx725n"
Apr 22 20:05:49.570953 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.570918    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-68cc5db7c4-gj5c7"]
Apr 22 20:05:49.571388 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.571291    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d1b8ae-bec1-4ba6-aeef-11b45aadf659" containerName="pull"
Apr 22 20:05:49.571388 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.571305    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d1b8ae-bec1-4ba6-aeef-11b45aadf659" containerName="pull"
Apr 22 20:05:49.571388 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.571322    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d1b8ae-bec1-4ba6-aeef-11b45aadf659" containerName="util"
Apr 22 20:05:49.571388 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.571331    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d1b8ae-bec1-4ba6-aeef-11b45aadf659" containerName="util"
Apr 22 20:05:49.571388 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.571341    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d1b8ae-bec1-4ba6-aeef-11b45aadf659" containerName="extract"
Apr 22 20:05:49.571388 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.571349    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d1b8ae-bec1-4ba6-aeef-11b45aadf659" containerName="extract"
Apr 22 20:05:49.571665 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.571441    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="45d1b8ae-bec1-4ba6-aeef-11b45aadf659" containerName="extract"
Apr 22 20:05:49.573951 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.573928    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-68cc5db7c4-gj5c7"
Apr 22 20:05:49.576332 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.576295    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Apr 22 20:05:49.576578 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.576564    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Apr 22 20:05:49.577681 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.577658    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Apr 22 20:05:49.577776 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.577717    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-cf52h\""
Apr 22 20:05:49.582392 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.582370    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-68cc5db7c4-gj5c7"]
Apr 22 20:05:49.684993 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.684966    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6de4e700-0c06-400b-845c-b372f8d40464-cert\") pod \"llmisvc-controller-manager-68cc5db7c4-gj5c7\" (UID: \"6de4e700-0c06-400b-845c-b372f8d40464\") " pod="kserve/llmisvc-controller-manager-68cc5db7c4-gj5c7"
Apr 22 20:05:49.685194 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.685029    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm9x7\" (UniqueName: \"kubernetes.io/projected/6de4e700-0c06-400b-845c-b372f8d40464-kube-api-access-wm9x7\") pod \"llmisvc-controller-manager-68cc5db7c4-gj5c7\" (UID: \"6de4e700-0c06-400b-845c-b372f8d40464\") " pod="kserve/llmisvc-controller-manager-68cc5db7c4-gj5c7"
Apr 22 20:05:49.785525 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.785499    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wm9x7\" (UniqueName: \"kubernetes.io/projected/6de4e700-0c06-400b-845c-b372f8d40464-kube-api-access-wm9x7\") pod \"llmisvc-controller-manager-68cc5db7c4-gj5c7\" (UID: \"6de4e700-0c06-400b-845c-b372f8d40464\") " pod="kserve/llmisvc-controller-manager-68cc5db7c4-gj5c7"
Apr 22 20:05:49.785670 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.785545    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6de4e700-0c06-400b-845c-b372f8d40464-cert\") pod \"llmisvc-controller-manager-68cc5db7c4-gj5c7\" (UID: \"6de4e700-0c06-400b-845c-b372f8d40464\") " pod="kserve/llmisvc-controller-manager-68cc5db7c4-gj5c7"
Apr 22 20:05:49.787915 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.787898    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6de4e700-0c06-400b-845c-b372f8d40464-cert\") pod \"llmisvc-controller-manager-68cc5db7c4-gj5c7\" (UID: \"6de4e700-0c06-400b-845c-b372f8d40464\") " pod="kserve/llmisvc-controller-manager-68cc5db7c4-gj5c7"
Apr 22 20:05:49.793383 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.793360    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm9x7\" (UniqueName: \"kubernetes.io/projected/6de4e700-0c06-400b-845c-b372f8d40464-kube-api-access-wm9x7\") pod \"llmisvc-controller-manager-68cc5db7c4-gj5c7\" (UID: \"6de4e700-0c06-400b-845c-b372f8d40464\") " pod="kserve/llmisvc-controller-manager-68cc5db7c4-gj5c7"
Apr 22 20:05:49.885204 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.885141    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-68cc5db7c4-gj5c7"
Apr 22 20:05:49.999514 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:49.999483    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-68cc5db7c4-gj5c7"]
Apr 22 20:05:50.002462 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:05:50.002401    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6de4e700_0c06_400b_845c_b372f8d40464.slice/crio-4d25e3c8b5837cd380e9db86fd3f712a88b1515f7f4fe3889e30421fc1bf7370 WatchSource:0}: Error finding container 4d25e3c8b5837cd380e9db86fd3f712a88b1515f7f4fe3889e30421fc1bf7370: Status 404 returned error can't find the container with id 4d25e3c8b5837cd380e9db86fd3f712a88b1515f7f4fe3889e30421fc1bf7370
Apr 22 20:05:50.498662 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:50.498623    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-68cc5db7c4-gj5c7" event={"ID":"6de4e700-0c06-400b-845c-b372f8d40464","Type":"ContainerStarted","Data":"4d25e3c8b5837cd380e9db86fd3f712a88b1515f7f4fe3889e30421fc1bf7370"}
Apr 22 20:05:52.505322 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:52.505285    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-68cc5db7c4-gj5c7" event={"ID":"6de4e700-0c06-400b-845c-b372f8d40464","Type":"ContainerStarted","Data":"fbe51be5870c5edcb19ad7ff14a53d313e1b22a773f02b9861f8fc1389313c5b"}
Apr 22 20:05:52.505710 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:52.505339    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-68cc5db7c4-gj5c7"
Apr 22 20:05:52.521747 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:05:52.521696    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-68cc5db7c4-gj5c7" podStartSLOduration=1.6759063250000001 podStartE2EDuration="3.521682965s" podCreationTimestamp="2026-04-22 20:05:49 +0000 UTC" firstStartedPulling="2026-04-22 20:05:50.004188196 +0000 UTC m=+501.421851819" lastFinishedPulling="2026-04-22 20:05:51.849964821 +0000 UTC m=+503.267628459" observedRunningTime="2026-04-22 20:05:52.520457566 +0000 UTC m=+503.938121212" watchObservedRunningTime="2026-04-22 20:05:52.521682965 +0000 UTC m=+503.939346611"
Apr 22 20:06:23.510318 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:06:23.510242    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-68cc5db7c4-gj5c7"
Apr 22 20:06:58.491000 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:06:58.490958    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-qs6b8"]
Apr 22 20:06:58.493858 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:06:58.493838    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-qs6b8"
Apr 22 20:06:58.496523 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:06:58.496505    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Apr 22 20:06:58.496632 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:06:58.496533    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-8qxvt\""
Apr 22 20:06:58.496802 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:06:58.496783    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-qs6b8"]
Apr 22 20:06:58.575327 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:06:58.575297    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sn54\" (UniqueName: \"kubernetes.io/projected/a8420c28-e439-4cc6-8d8b-3d960d64a87e-kube-api-access-6sn54\") pod \"odh-model-controller-696fc77849-qs6b8\" (UID: \"a8420c28-e439-4cc6-8d8b-3d960d64a87e\") " pod="kserve/odh-model-controller-696fc77849-qs6b8"
Apr 22 20:06:58.575460 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:06:58.575339    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8420c28-e439-4cc6-8d8b-3d960d64a87e-cert\") pod \"odh-model-controller-696fc77849-qs6b8\" (UID: \"a8420c28-e439-4cc6-8d8b-3d960d64a87e\") " pod="kserve/odh-model-controller-696fc77849-qs6b8"
Apr 22 20:06:58.676109 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:06:58.676083    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6sn54\" (UniqueName: \"kubernetes.io/projected/a8420c28-e439-4cc6-8d8b-3d960d64a87e-kube-api-access-6sn54\") pod \"odh-model-controller-696fc77849-qs6b8\" (UID: \"a8420c28-e439-4cc6-8d8b-3d960d64a87e\") " pod="kserve/odh-model-controller-696fc77849-qs6b8"
Apr 22 20:06:58.676226 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:06:58.676131    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8420c28-e439-4cc6-8d8b-3d960d64a87e-cert\") pod \"odh-model-controller-696fc77849-qs6b8\" (UID: \"a8420c28-e439-4cc6-8d8b-3d960d64a87e\") " pod="kserve/odh-model-controller-696fc77849-qs6b8"
Apr 22 20:06:58.678485 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:06:58.678458    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8420c28-e439-4cc6-8d8b-3d960d64a87e-cert\") pod \"odh-model-controller-696fc77849-qs6b8\" (UID: \"a8420c28-e439-4cc6-8d8b-3d960d64a87e\") " pod="kserve/odh-model-controller-696fc77849-qs6b8"
Apr 22 20:06:58.684498 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:06:58.684469    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sn54\" (UniqueName: \"kubernetes.io/projected/a8420c28-e439-4cc6-8d8b-3d960d64a87e-kube-api-access-6sn54\") pod \"odh-model-controller-696fc77849-qs6b8\" (UID: \"a8420c28-e439-4cc6-8d8b-3d960d64a87e\") " pod="kserve/odh-model-controller-696fc77849-qs6b8"
Apr 22 20:06:58.805780 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:06:58.805711    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-qs6b8"
Apr 22 20:06:58.919203 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:06:58.919178    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-qs6b8"]
Apr 22 20:06:58.921727 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:06:58.921691    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8420c28_e439_4cc6_8d8b_3d960d64a87e.slice/crio-da490283c23b1f9d04c45006520443a561ee68d377d29b752c19c8dadaeba164 WatchSource:0}: Error finding container da490283c23b1f9d04c45006520443a561ee68d377d29b752c19c8dadaeba164: Status 404 returned error can't find the container with id da490283c23b1f9d04c45006520443a561ee68d377d29b752c19c8dadaeba164
Apr 22 20:06:59.695370 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:06:59.695320    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-qs6b8" event={"ID":"a8420c28-e439-4cc6-8d8b-3d960d64a87e","Type":"ContainerStarted","Data":"da490283c23b1f9d04c45006520443a561ee68d377d29b752c19c8dadaeba164"}
Apr 22 20:07:01.702150 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:01.702114    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-qs6b8" event={"ID":"a8420c28-e439-4cc6-8d8b-3d960d64a87e","Type":"ContainerStarted","Data":"9a656f9233d0f568f69bc9d7ea8f4593729cfd388a1839099fcb0f77c04bc856"}
Apr 22 20:07:01.702568 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:01.702199    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-qs6b8"
Apr 22 20:07:01.717370 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:01.717315    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-qs6b8" podStartSLOduration=1.084403937 podStartE2EDuration="3.717300715s" podCreationTimestamp="2026-04-22 20:06:58 +0000 UTC" firstStartedPulling="2026-04-22 20:06:58.923001653 +0000 UTC m=+570.340665277" lastFinishedPulling="2026-04-22 20:07:01.555898428 +0000 UTC m=+572.973562055" observedRunningTime="2026-04-22 20:07:01.716669112 +0000 UTC m=+573.134332758" watchObservedRunningTime="2026-04-22 20:07:01.717300715 +0000 UTC m=+573.134964362"
Apr 22 20:07:12.708087 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:12.708054    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-qs6b8"
Apr 22 20:07:13.511343 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:13.511313    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-4snb2"]
Apr 22 20:07:13.514443 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:13.514427    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4snb2"
Apr 22 20:07:13.516878 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:13.516856    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Apr 22 20:07:13.517002 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:13.516965    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q9sqg\""
Apr 22 20:07:13.521784 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:13.521762    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-4snb2"]
Apr 22 20:07:13.587475 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:13.587449    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65bgf\" (UniqueName: \"kubernetes.io/projected/68b9a15a-d6dc-4c5f-8d62-ead9a73bab96-kube-api-access-65bgf\") pod \"s3-init-4snb2\" (UID: \"68b9a15a-d6dc-4c5f-8d62-ead9a73bab96\") " pod="kserve/s3-init-4snb2"
Apr 22 20:07:13.688745 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:13.688713    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-65bgf\" (UniqueName: \"kubernetes.io/projected/68b9a15a-d6dc-4c5f-8d62-ead9a73bab96-kube-api-access-65bgf\") pod \"s3-init-4snb2\" (UID: \"68b9a15a-d6dc-4c5f-8d62-ead9a73bab96\") " pod="kserve/s3-init-4snb2"
Apr 22 20:07:13.697426 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:13.697395    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65bgf\" (UniqueName: \"kubernetes.io/projected/68b9a15a-d6dc-4c5f-8d62-ead9a73bab96-kube-api-access-65bgf\") pod \"s3-init-4snb2\" (UID: \"68b9a15a-d6dc-4c5f-8d62-ead9a73bab96\") " pod="kserve/s3-init-4snb2"
Apr 22 20:07:13.832993 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:13.832926    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4snb2"
Apr 22 20:07:13.952281 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:13.952254    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-4snb2"]
Apr 22 20:07:13.955981 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:07:13.955948    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68b9a15a_d6dc_4c5f_8d62_ead9a73bab96.slice/crio-7d65e849adf732734d63445d1aa28171d611e6159acec8cac148dca5bb828b1b WatchSource:0}: Error finding container 7d65e849adf732734d63445d1aa28171d611e6159acec8cac148dca5bb828b1b: Status 404 returned error can't find the container with id 7d65e849adf732734d63445d1aa28171d611e6159acec8cac148dca5bb828b1b
Apr 22 20:07:14.739337 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:14.739297    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4snb2" event={"ID":"68b9a15a-d6dc-4c5f-8d62-ead9a73bab96","Type":"ContainerStarted","Data":"7d65e849adf732734d63445d1aa28171d611e6159acec8cac148dca5bb828b1b"}
Apr 22 20:07:18.758912 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:18.758868    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4snb2" event={"ID":"68b9a15a-d6dc-4c5f-8d62-ead9a73bab96","Type":"ContainerStarted","Data":"a94a25e9527b174393240eb74a65af2c2ccb1996d25450af2a49e9834c47400d"}
Apr 22 20:07:18.773832 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:18.773780    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-4snb2" podStartSLOduration=1.332801966 podStartE2EDuration="5.773767546s" podCreationTimestamp="2026-04-22 20:07:13 +0000 UTC" firstStartedPulling="2026-04-22 20:07:13.958116644 +0000 UTC m=+585.375780268" lastFinishedPulling="2026-04-22 20:07:18.39908222 +0000 UTC m=+589.816745848" observedRunningTime="2026-04-22 20:07:18.772873334 +0000 UTC m=+590.190536978" watchObservedRunningTime="2026-04-22 20:07:18.773767546 +0000 UTC m=+590.191431192"
Apr 22 20:07:21.769045 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:21.769013    2574 generic.go:358] "Generic (PLEG): container finished" podID="68b9a15a-d6dc-4c5f-8d62-ead9a73bab96" containerID="a94a25e9527b174393240eb74a65af2c2ccb1996d25450af2a49e9834c47400d" exitCode=0
Apr 22 20:07:21.769405 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:21.769063    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4snb2" event={"ID":"68b9a15a-d6dc-4c5f-8d62-ead9a73bab96","Type":"ContainerDied","Data":"a94a25e9527b174393240eb74a65af2c2ccb1996d25450af2a49e9834c47400d"}
Apr 22 20:07:22.896407 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:22.896384    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4snb2"
Apr 22 20:07:22.962057 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:22.962020    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65bgf\" (UniqueName: \"kubernetes.io/projected/68b9a15a-d6dc-4c5f-8d62-ead9a73bab96-kube-api-access-65bgf\") pod \"68b9a15a-d6dc-4c5f-8d62-ead9a73bab96\" (UID: \"68b9a15a-d6dc-4c5f-8d62-ead9a73bab96\") "
Apr 22 20:07:22.964299 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:22.964272    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b9a15a-d6dc-4c5f-8d62-ead9a73bab96-kube-api-access-65bgf" (OuterVolumeSpecName: "kube-api-access-65bgf") pod "68b9a15a-d6dc-4c5f-8d62-ead9a73bab96" (UID: "68b9a15a-d6dc-4c5f-8d62-ead9a73bab96"). InnerVolumeSpecName "kube-api-access-65bgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:07:23.063636 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:23.063536    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65bgf\" (UniqueName: \"kubernetes.io/projected/68b9a15a-d6dc-4c5f-8d62-ead9a73bab96-kube-api-access-65bgf\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:07:23.776137 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:23.776102    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4snb2" event={"ID":"68b9a15a-d6dc-4c5f-8d62-ead9a73bab96","Type":"ContainerDied","Data":"7d65e849adf732734d63445d1aa28171d611e6159acec8cac148dca5bb828b1b"}
Apr 22 20:07:23.776137 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:23.776133    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d65e849adf732734d63445d1aa28171d611e6159acec8cac148dca5bb828b1b"
Apr 22 20:07:23.776388 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:23.776151    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4snb2"
Apr 22 20:07:32.669562 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:32.669522    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww"]
Apr 22 20:07:32.670033 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:32.669935    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68b9a15a-d6dc-4c5f-8d62-ead9a73bab96" containerName="s3-init"
Apr 22 20:07:32.670033 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:32.669949    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="68b9a15a-d6dc-4c5f-8d62-ead9a73bab96" containerName="s3-init"
Apr 22 20:07:32.670144 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:32.670039    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="68b9a15a-d6dc-4c5f-8d62-ead9a73bab96" containerName="s3-init"
Apr 22 20:07:32.672915 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:32.672890    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww"
Apr 22 20:07:32.675363 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:32.675340    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4jdsb\""
Apr 22 20:07:32.679351 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:32.679322    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww"]
Apr 22 20:07:32.685380 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:32.685362    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww"
Apr 22 20:07:32.821284 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:32.821019    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww"]
Apr 22 20:07:32.824108 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:07:32.824076    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdfdc40_6ef8_4be0_a83e_6db6bfddb155.slice/crio-b9202712ac1ef8076e00de3806d785c65f3e7ac641e2848e804ead287a4f8c76 WatchSource:0}: Error finding container b9202712ac1ef8076e00de3806d785c65f3e7ac641e2848e804ead287a4f8c76: Status 404 returned error can't find the container with id b9202712ac1ef8076e00de3806d785c65f3e7ac641e2848e804ead287a4f8c76
Apr 22 20:07:32.874084 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:32.874053    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d"]
Apr 22 20:07:32.876891 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:32.876870    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d"
Apr 22 20:07:32.887395 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:32.887365    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d"]
Apr 22 20:07:32.942280 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:32.942248    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6984557478-8rb9d\" (UID: \"2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d"
Apr 22 20:07:33.045435 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:33.043057    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6984557478-8rb9d\" (UID: \"2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d"
Apr 22 20:07:33.045435 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:33.043489    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6984557478-8rb9d\" (UID: \"2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d"
Apr 22 20:07:33.188760 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:33.188729    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d"
Apr 22 20:07:33.328495 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:33.328291    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d"]
Apr 22 20:07:33.331054 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:07:33.331020    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2596b2c6_6621_4c8b_b3ab_e4e4e9a7813a.slice/crio-9c36ea7f3529a96fe64b6312682c04c987914cbb33c62353d47145a98d0f0ae8 WatchSource:0}: Error finding container 9c36ea7f3529a96fe64b6312682c04c987914cbb33c62353d47145a98d0f0ae8: Status 404 returned error can't find the container with id 9c36ea7f3529a96fe64b6312682c04c987914cbb33c62353d47145a98d0f0ae8
Apr 22 20:07:33.812662 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:33.812626    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d" event={"ID":"2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a","Type":"ContainerStarted","Data":"9c36ea7f3529a96fe64b6312682c04c987914cbb33c62353d47145a98d0f0ae8"}
Apr 22 20:07:33.814691 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:33.814662    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww" event={"ID":"4bdfdc40-6ef8-4be0-a83e-6db6bfddb155","Type":"ContainerStarted","Data":"b9202712ac1ef8076e00de3806d785c65f3e7ac641e2848e804ead287a4f8c76"}
Apr 22 20:07:47.863782 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:47.863741    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww" event={"ID":"4bdfdc40-6ef8-4be0-a83e-6db6bfddb155","Type":"ContainerStarted","Data":"8c3ae5244706c48b697ad9236648c8f34512a7f79dc0f3a949558647179bd1fc"}
Apr 22 20:07:47.864186 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:47.864006    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww"
Apr 22 20:07:47.865326 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:47.865284    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww" podUID="4bdfdc40-6ef8-4be0-a83e-6db6bfddb155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Apr 22 20:07:47.865488 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:47.865455    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d" event={"ID":"2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a","Type":"ContainerStarted","Data":"5d09fda83f609e00de6a308e15e539537e2ab3452ef8bb29e67ec90753ccd857"}
Apr 22 20:07:47.878242 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:47.878196    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww" podStartSLOduration=1.802933039 podStartE2EDuration="15.878181112s" podCreationTimestamp="2026-04-22 20:07:32 +0000 UTC" firstStartedPulling="2026-04-22 20:07:32.826669062 +0000 UTC m=+604.244332688" lastFinishedPulling="2026-04-22 20:07:46.901917122 +0000 UTC m=+618.319580761" observedRunningTime="2026-04-22 20:07:47.877696799 +0000 UTC m=+619.295360473" watchObservedRunningTime="2026-04-22 20:07:47.878181112 +0000 UTC m=+619.295844769"
Apr 22 20:07:48.869378 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:48.869339    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww" podUID="4bdfdc40-6ef8-4be0-a83e-6db6bfddb155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Apr 22 20:07:50.876580 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:50.876544    2574 generic.go:358] "Generic (PLEG): container finished" podID="2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a" containerID="5d09fda83f609e00de6a308e15e539537e2ab3452ef8bb29e67ec90753ccd857" exitCode=0
Apr 22 20:07:50.876887 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:50.876646    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d" event={"ID":"2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a","Type":"ContainerDied","Data":"5d09fda83f609e00de6a308e15e539537e2ab3452ef8bb29e67ec90753ccd857"}
Apr 22 20:07:57.905637 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:57.905605    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d" event={"ID":"2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a","Type":"ContainerStarted","Data":"c9c5f209ab51ca42b609f1982a38e9dfce868621090b24b245db61da4fdd0e43"}
Apr 22 20:07:57.906082 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:57.905907    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d"
Apr 22 20:07:57.907126 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:57.907100    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d" podUID="2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Apr 22 20:07:57.922875 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:57.922832    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d" podStartSLOduration=2.196197131 podStartE2EDuration="25.922818805s" podCreationTimestamp="2026-04-22 20:07:32 +0000 UTC" firstStartedPulling="2026-04-22 20:07:33.333453782 +0000 UTC m=+604.751117412" lastFinishedPulling="2026-04-22 20:07:57.060075462 +0000 UTC m=+628.477739086" observedRunningTime="2026-04-22 20:07:57.921656683 +0000 UTC m=+629.339320354" watchObservedRunningTime="2026-04-22 20:07:57.922818805 +0000 UTC m=+629.340482450"
Apr 22 20:07:58.870075 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:58.870036    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww" podUID="4bdfdc40-6ef8-4be0-a83e-6db6bfddb155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Apr 22 20:07:58.909217 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:07:58.909178    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d" podUID="2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Apr 22 20:08:08.869891 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:08.869845    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww" podUID="4bdfdc40-6ef8-4be0-a83e-6db6bfddb155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Apr 22 20:08:08.909530 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:08.909495    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d" podUID="2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Apr 22 20:08:18.869699 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:18.869651    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww" podUID="4bdfdc40-6ef8-4be0-a83e-6db6bfddb155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Apr 22 20:08:18.909964 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:18.909920    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d" podUID="2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Apr 22 20:08:28.870129 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:28.870085    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww" podUID="4bdfdc40-6ef8-4be0-a83e-6db6bfddb155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Apr 22 20:08:28.909163 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:28.909130    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d" podUID="2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Apr 22 20:08:38.870098 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:38.870065    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww"
Apr 22 20:08:38.909551 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:38.909510    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d" podUID="2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Apr 22 20:08:48.909211 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:48.909168    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d" podUID="2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Apr 22 20:08:52.755940 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:52.755910    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s"]
Apr 22 20:08:52.758452 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:52.758433    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s"
Apr 22 20:08:52.760938 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:52.760914    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-bee44-serving-cert\""
Apr 22 20:08:52.760938 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:52.760931    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-bee44-kube-rbac-proxy-sar-config\""
Apr 22 20:08:52.761101 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:52.760980    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 22 20:08:52.768391 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:52.768372    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s"]
Apr 22 20:08:52.868588 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:52.868562    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/020d2e80-1a3d-4aa3-a220-9c626491fefd-openshift-service-ca-bundle\") pod \"switch-graph-bee44-69f6ff6cd8-74v7s\" (UID: \"020d2e80-1a3d-4aa3-a220-9c626491fefd\") " pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s"
Apr 22 20:08:52.868723 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:52.868657    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/020d2e80-1a3d-4aa3-a220-9c626491fefd-proxy-tls\") pod \"switch-graph-bee44-69f6ff6cd8-74v7s\" (UID: \"020d2e80-1a3d-4aa3-a220-9c626491fefd\") " pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s"
Apr 22 20:08:52.969565 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:52.969540    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/020d2e80-1a3d-4aa3-a220-9c626491fefd-proxy-tls\") pod \"switch-graph-bee44-69f6ff6cd8-74v7s\" (UID: \"020d2e80-1a3d-4aa3-a220-9c626491fefd\") " pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s"
Apr 22 20:08:52.969676 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:52.969579    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/020d2e80-1a3d-4aa3-a220-9c626491fefd-openshift-service-ca-bundle\") pod \"switch-graph-bee44-69f6ff6cd8-74v7s\" (UID: \"020d2e80-1a3d-4aa3-a220-9c626491fefd\") " pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s"
Apr 22 20:08:52.969775 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:08:52.969703    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-bee44-serving-cert: secret "switch-graph-bee44-serving-cert" not found
Apr 22 20:08:52.969845 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:08:52.969782    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/020d2e80-1a3d-4aa3-a220-9c626491fefd-proxy-tls podName:020d2e80-1a3d-4aa3-a220-9c626491fefd nodeName:}" failed. No retries permitted until 2026-04-22 20:08:53.469759247 +0000 UTC m=+684.887422878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/020d2e80-1a3d-4aa3-a220-9c626491fefd-proxy-tls") pod "switch-graph-bee44-69f6ff6cd8-74v7s" (UID: "020d2e80-1a3d-4aa3-a220-9c626491fefd") : secret "switch-graph-bee44-serving-cert" not found
Apr 22 20:08:52.970161 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:52.970141    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/020d2e80-1a3d-4aa3-a220-9c626491fefd-openshift-service-ca-bundle\") pod \"switch-graph-bee44-69f6ff6cd8-74v7s\" (UID: \"020d2e80-1a3d-4aa3-a220-9c626491fefd\") " pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s"
Apr 22 20:08:53.472683 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:53.472647    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/020d2e80-1a3d-4aa3-a220-9c626491fefd-proxy-tls\") pod \"switch-graph-bee44-69f6ff6cd8-74v7s\" (UID: \"020d2e80-1a3d-4aa3-a220-9c626491fefd\") " pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s"
Apr 22 20:08:53.475071 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:53.475048    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/020d2e80-1a3d-4aa3-a220-9c626491fefd-proxy-tls\") pod \"switch-graph-bee44-69f6ff6cd8-74v7s\" (UID: \"020d2e80-1a3d-4aa3-a220-9c626491fefd\") " pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s"
Apr 22 20:08:53.668964 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:53.668938    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s"
Apr 22 20:08:53.783644 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:53.783533    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s"]
Apr 22 20:08:53.786376 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:08:53.786345    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020d2e80_1a3d_4aa3_a220_9c626491fefd.slice/crio-07cfac1cfbb7ba065a38fb8b3dafc9101c1a53311d0169dc02bbf3072bad084a WatchSource:0}: Error finding container 07cfac1cfbb7ba065a38fb8b3dafc9101c1a53311d0169dc02bbf3072bad084a: Status 404 returned error can't find the container with id 07cfac1cfbb7ba065a38fb8b3dafc9101c1a53311d0169dc02bbf3072bad084a
Apr 22 20:08:54.070581 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:54.070496    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s" event={"ID":"020d2e80-1a3d-4aa3-a220-9c626491fefd","Type":"ContainerStarted","Data":"07cfac1cfbb7ba065a38fb8b3dafc9101c1a53311d0169dc02bbf3072bad084a"}
Apr 22 20:08:57.081282 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:57.081237    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s" event={"ID":"020d2e80-1a3d-4aa3-a220-9c626491fefd","Type":"ContainerStarted","Data":"d1052488260ddf30f5443d529ce2ca208f61200378e247d05ac17e83bda93072"}
Apr 22 20:08:57.081746 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:57.081356    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s"
Apr 22 20:08:57.098027 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:57.097979    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s" podStartSLOduration=2.430698773 podStartE2EDuration="5.097966368s" podCreationTimestamp="2026-04-22 20:08:52 +0000 UTC" firstStartedPulling="2026-04-22 20:08:53.788078547 +0000 UTC m=+685.205742171" lastFinishedPulling="2026-04-22 20:08:56.455346142 +0000 UTC m=+687.873009766" observedRunningTime="2026-04-22 20:08:57.096152802 +0000 UTC m=+688.513816452" watchObservedRunningTime="2026-04-22 20:08:57.097966368 +0000 UTC m=+688.515630013"
Apr 22 20:08:58.909452 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:08:58.909392    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d" podUID="2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Apr 22 20:09:03.088979 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:03.088951    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s"
Apr 22 20:09:05.160899 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:05.160870    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d"
Apr 22 20:09:06.963560 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:06.963524    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s"]
Apr 22 20:09:06.964023 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:06.963759    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s" podUID="020d2e80-1a3d-4aa3-a220-9c626491fefd" containerName="switch-graph-bee44" containerID="cri-o://d1052488260ddf30f5443d529ce2ca208f61200378e247d05ac17e83bda93072" gracePeriod=30
Apr 22 20:09:07.080392 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:07.080363    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww"]
Apr 22 20:09:07.080626 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:07.080604    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww" podUID="4bdfdc40-6ef8-4be0-a83e-6db6bfddb155" containerName="kserve-container" containerID="cri-o://8c3ae5244706c48b697ad9236648c8f34512a7f79dc0f3a949558647179bd1fc" gracePeriod=30
Apr 22 20:09:07.132234 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:07.132196    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt"]
Apr 22 20:09:07.135571 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:07.135549    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt"
Apr 22 20:09:07.144447 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:07.144401    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt"]
Apr 22 20:09:07.150172 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:07.150153    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt"
Apr 22 20:09:07.278696 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:07.278668    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt"]
Apr 22 20:09:07.281359 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:09:07.281318    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d922f9_3c12_411c_b055_ff26058bdd06.slice/crio-adf5f6a3a28f2ac2cd79ccabdb59ce5e8ac585c96553c794cbc1422a0bd8d1e9 WatchSource:0}: Error finding container adf5f6a3a28f2ac2cd79ccabdb59ce5e8ac585c96553c794cbc1422a0bd8d1e9: Status 404 returned error can't find the container with id adf5f6a3a28f2ac2cd79ccabdb59ce5e8ac585c96553c794cbc1422a0bd8d1e9
Apr 22 20:09:08.088630 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:08.088592    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s" podUID="020d2e80-1a3d-4aa3-a220-9c626491fefd" containerName="switch-graph-bee44" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:09:08.114195 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:08.114165    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt" event={"ID":"f7d922f9-3c12-411c-b055-ff26058bdd06","Type":"ContainerStarted","Data":"fb646155cde324315b6d2be3eafe4ce43519bd7e37b5316193ab05fff436c4b7"}
Apr 22 20:09:08.114334 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:08.114200    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt" event={"ID":"f7d922f9-3c12-411c-b055-ff26058bdd06","Type":"ContainerStarted","Data":"adf5f6a3a28f2ac2cd79ccabdb59ce5e8ac585c96553c794cbc1422a0bd8d1e9"}
Apr 22 20:09:08.114401 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:08.114380    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt"
Apr 22 20:09:08.115649 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:08.115627    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt" podUID="f7d922f9-3c12-411c-b055-ff26058bdd06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Apr 22 20:09:08.127810 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:08.127767    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt" podStartSLOduration=1.127752354 podStartE2EDuration="1.127752354s" podCreationTimestamp="2026-04-22 20:09:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:09:08.127716659 +0000 UTC m=+699.545380316" watchObservedRunningTime="2026-04-22 20:09:08.127752354 +0000 UTC m=+699.545416002"
Apr 22 20:09:08.870125 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:08.870089    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww" podUID="4bdfdc40-6ef8-4be0-a83e-6db6bfddb155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Apr 22 20:09:09.117140 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:09.117100    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt" podUID="f7d922f9-3c12-411c-b055-ff26058bdd06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Apr 22 20:09:09.923804 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:09.923777    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww"
Apr 22 20:09:10.120822 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:10.120734    2574 generic.go:358] "Generic (PLEG): container finished" podID="4bdfdc40-6ef8-4be0-a83e-6db6bfddb155" containerID="8c3ae5244706c48b697ad9236648c8f34512a7f79dc0f3a949558647179bd1fc" exitCode=0
Apr 22 20:09:10.121193 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:10.120822    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww" event={"ID":"4bdfdc40-6ef8-4be0-a83e-6db6bfddb155","Type":"ContainerDied","Data":"8c3ae5244706c48b697ad9236648c8f34512a7f79dc0f3a949558647179bd1fc"}
Apr 22 20:09:10.121193 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:10.120832    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww"
Apr 22 20:09:10.121193 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:10.120876    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww" event={"ID":"4bdfdc40-6ef8-4be0-a83e-6db6bfddb155","Type":"ContainerDied","Data":"b9202712ac1ef8076e00de3806d785c65f3e7ac641e2848e804ead287a4f8c76"}
Apr 22 20:09:10.121193 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:10.120897    2574 scope.go:117] "RemoveContainer" containerID="8c3ae5244706c48b697ad9236648c8f34512a7f79dc0f3a949558647179bd1fc"
Apr 22 20:09:10.128967 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:10.128946    2574 scope.go:117] "RemoveContainer" containerID="8c3ae5244706c48b697ad9236648c8f34512a7f79dc0f3a949558647179bd1fc"
Apr 22 20:09:10.129261 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:09:10.129236    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c3ae5244706c48b697ad9236648c8f34512a7f79dc0f3a949558647179bd1fc\": container with ID starting with 8c3ae5244706c48b697ad9236648c8f34512a7f79dc0f3a949558647179bd1fc not found: ID does not exist" containerID="8c3ae5244706c48b697ad9236648c8f34512a7f79dc0f3a949558647179bd1fc"
Apr 22 20:09:10.129341 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:10.129270    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c3ae5244706c48b697ad9236648c8f34512a7f79dc0f3a949558647179bd1fc"} err="failed to get container status \"8c3ae5244706c48b697ad9236648c8f34512a7f79dc0f3a949558647179bd1fc\": rpc error: code = NotFound desc = could not find container \"8c3ae5244706c48b697ad9236648c8f34512a7f79dc0f3a949558647179bd1fc\": container with ID starting with 8c3ae5244706c48b697ad9236648c8f34512a7f79dc0f3a949558647179bd1fc not found: ID does not exist"
Apr 22 20:09:10.141013 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:10.140993    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww"]
Apr 22 20:09:10.143555 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:10.143537    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bee44-predictor-68957dbb9d-pk7ww"]
Apr 22 20:09:11.160008 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:11.159971    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bdfdc40-6ef8-4be0-a83e-6db6bfddb155" path="/var/lib/kubelet/pods/4bdfdc40-6ef8-4be0-a83e-6db6bfddb155/volumes"
Apr 22 20:09:13.088639 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:13.088606    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s" podUID="020d2e80-1a3d-4aa3-a220-9c626491fefd" containerName="switch-graph-bee44" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:09:18.088606 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:18.088526    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s" podUID="020d2e80-1a3d-4aa3-a220-9c626491fefd" containerName="switch-graph-bee44" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:09:18.088963 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:18.088633    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s"
Apr 22 20:09:19.118026 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:19.117985    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt" podUID="f7d922f9-3c12-411c-b055-ff26058bdd06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Apr 22 20:09:23.087642 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:23.087597    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s" podUID="020d2e80-1a3d-4aa3-a220-9c626491fefd" containerName="switch-graph-bee44" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:09:28.088116 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:28.088080    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s" podUID="020d2e80-1a3d-4aa3-a220-9c626491fefd" containerName="switch-graph-bee44" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:09:29.118013 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:29.117979    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt" podUID="f7d922f9-3c12-411c-b055-ff26058bdd06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Apr 22 20:09:32.737606 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:32.737576    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv"]
Apr 22 20:09:32.738013 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:32.737914    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bdfdc40-6ef8-4be0-a83e-6db6bfddb155" containerName="kserve-container"
Apr 22 20:09:32.738013 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:32.737930    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bdfdc40-6ef8-4be0-a83e-6db6bfddb155" containerName="kserve-container"
Apr 22 20:09:32.738108 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:32.738027    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="4bdfdc40-6ef8-4be0-a83e-6db6bfddb155" containerName="kserve-container"
Apr 22 20:09:32.743766 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:32.743745    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv"
Apr 22 20:09:32.746577 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:32.746499    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Apr 22 20:09:32.746577 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:32.746534    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Apr 22 20:09:32.747588 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:32.747564    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv"]
Apr 22 20:09:32.872587 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:32.872546    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/444625ef-d103-42c3-acaf-b3e82a9d48d0-proxy-tls\") pod \"model-chainer-6f87c58cc9-5tnmv\" (UID: \"444625ef-d103-42c3-acaf-b3e82a9d48d0\") " pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv"
Apr 22 20:09:32.872741 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:32.872623    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/444625ef-d103-42c3-acaf-b3e82a9d48d0-openshift-service-ca-bundle\") pod \"model-chainer-6f87c58cc9-5tnmv\" (UID: \"444625ef-d103-42c3-acaf-b3e82a9d48d0\") " pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv"
Apr 22 20:09:32.973920 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:32.973893    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/444625ef-d103-42c3-acaf-b3e82a9d48d0-proxy-tls\") pod \"model-chainer-6f87c58cc9-5tnmv\" (UID: \"444625ef-d103-42c3-acaf-b3e82a9d48d0\") " pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv"
Apr 22 20:09:32.974045 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:32.973934    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/444625ef-d103-42c3-acaf-b3e82a9d48d0-openshift-service-ca-bundle\") pod \"model-chainer-6f87c58cc9-5tnmv\" (UID: \"444625ef-d103-42c3-acaf-b3e82a9d48d0\") " pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv"
Apr 22 20:09:32.974504 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:32.974487    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/444625ef-d103-42c3-acaf-b3e82a9d48d0-openshift-service-ca-bundle\") pod \"model-chainer-6f87c58cc9-5tnmv\" (UID: \"444625ef-d103-42c3-acaf-b3e82a9d48d0\") " pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv"
Apr 22 20:09:32.976372 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:32.976355    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/444625ef-d103-42c3-acaf-b3e82a9d48d0-proxy-tls\") pod \"model-chainer-6f87c58cc9-5tnmv\" (UID: \"444625ef-d103-42c3-acaf-b3e82a9d48d0\") " pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv"
Apr 22 20:09:33.056145 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:33.056085    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv"
Apr 22 20:09:33.088520 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:33.088486    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s" podUID="020d2e80-1a3d-4aa3-a220-9c626491fefd" containerName="switch-graph-bee44" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:09:33.173020 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:33.172983    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv"]
Apr 22 20:09:33.178460 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:33.178405    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 22 20:09:33.187402 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:33.187377    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv" event={"ID":"444625ef-d103-42c3-acaf-b3e82a9d48d0","Type":"ContainerStarted","Data":"ee13e1e606a49ea7098f69375f4b17670b79353439072daf4b8bcc65de482737"}
Apr 22 20:09:34.190997 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:34.190959    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv" event={"ID":"444625ef-d103-42c3-acaf-b3e82a9d48d0","Type":"ContainerStarted","Data":"e6c5d7087d8bcbcdb8ecc3226fcdadd701da0953db5c0041e329c92f260afe41"}
Apr 22 20:09:34.191443 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:34.191038    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv"
Apr 22 20:09:34.205076 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:34.205039    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv" podStartSLOduration=2.205026205 podStartE2EDuration="2.205026205s" podCreationTimestamp="2026-04-22 20:09:32 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:09:34.204596503 +0000 UTC m=+725.622260176" watchObservedRunningTime="2026-04-22 20:09:34.205026205 +0000 UTC m=+725.622689851"
Apr 22 20:09:37.008062 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:09:37.008032    2574 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020d2e80_1a3d_4aa3_a220_9c626491fefd.slice/crio-conmon-d1052488260ddf30f5443d529ce2ca208f61200378e247d05ac17e83bda93072.scope\": RecentStats: unable to find data in memory cache]"
Apr 22 20:09:37.106829 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:37.106809    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s"
Apr 22 20:09:37.200121 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:37.200092    2574 generic.go:358] "Generic (PLEG): container finished" podID="020d2e80-1a3d-4aa3-a220-9c626491fefd" containerID="d1052488260ddf30f5443d529ce2ca208f61200378e247d05ac17e83bda93072" exitCode=0
Apr 22 20:09:37.200239 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:37.200138    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s"
Apr 22 20:09:37.200239 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:37.200172    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s" event={"ID":"020d2e80-1a3d-4aa3-a220-9c626491fefd","Type":"ContainerDied","Data":"d1052488260ddf30f5443d529ce2ca208f61200378e247d05ac17e83bda93072"}
Apr 22 20:09:37.200239 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:37.200207    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s" event={"ID":"020d2e80-1a3d-4aa3-a220-9c626491fefd","Type":"ContainerDied","Data":"07cfac1cfbb7ba065a38fb8b3dafc9101c1a53311d0169dc02bbf3072bad084a"}
Apr 22 20:09:37.200239 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:37.200221    2574 scope.go:117] "RemoveContainer" containerID="d1052488260ddf30f5443d529ce2ca208f61200378e247d05ac17e83bda93072"
Apr 22 20:09:37.206495 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:37.206473    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/020d2e80-1a3d-4aa3-a220-9c626491fefd-proxy-tls\") pod \"020d2e80-1a3d-4aa3-a220-9c626491fefd\" (UID: \"020d2e80-1a3d-4aa3-a220-9c626491fefd\") "
Apr 22 20:09:37.206605 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:37.206589    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/020d2e80-1a3d-4aa3-a220-9c626491fefd-openshift-service-ca-bundle\") pod \"020d2e80-1a3d-4aa3-a220-9c626491fefd\" (UID: \"020d2e80-1a3d-4aa3-a220-9c626491fefd\") "
Apr 22 20:09:37.206959 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:37.206937    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020d2e80-1a3d-4aa3-a220-9c626491fefd-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "020d2e80-1a3d-4aa3-a220-9c626491fefd" (UID: "020d2e80-1a3d-4aa3-a220-9c626491fefd"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:09:37.207529 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:37.207507    2574 scope.go:117] "RemoveContainer" containerID="d1052488260ddf30f5443d529ce2ca208f61200378e247d05ac17e83bda93072"
Apr 22 20:09:37.207772 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:09:37.207755    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1052488260ddf30f5443d529ce2ca208f61200378e247d05ac17e83bda93072\": container with ID starting with d1052488260ddf30f5443d529ce2ca208f61200378e247d05ac17e83bda93072 not found: ID does not exist" containerID="d1052488260ddf30f5443d529ce2ca208f61200378e247d05ac17e83bda93072"
Apr 22 20:09:37.207836 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:37.207778    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1052488260ddf30f5443d529ce2ca208f61200378e247d05ac17e83bda93072"} err="failed to get container status \"d1052488260ddf30f5443d529ce2ca208f61200378e247d05ac17e83bda93072\": rpc error: code = NotFound desc = could not find container \"d1052488260ddf30f5443d529ce2ca208f61200378e247d05ac17e83bda93072\": container with ID starting with d1052488260ddf30f5443d529ce2ca208f61200378e247d05ac17e83bda93072 not found: ID does not exist"
Apr 22 20:09:37.208694 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:37.208675    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/020d2e80-1a3d-4aa3-a220-9c626491fefd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "020d2e80-1a3d-4aa3-a220-9c626491fefd" (UID: "020d2e80-1a3d-4aa3-a220-9c626491fefd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:09:37.307250 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:37.307178    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/020d2e80-1a3d-4aa3-a220-9c626491fefd-openshift-service-ca-bundle\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:09:37.307250 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:37.307219    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/020d2e80-1a3d-4aa3-a220-9c626491fefd-proxy-tls\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:09:37.524228 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:37.524207    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s"]
Apr 22 20:09:37.527576 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:37.527556    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bee44-69f6ff6cd8-74v7s"]
Apr 22 20:09:39.117956 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:39.117918    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt" podUID="f7d922f9-3c12-411c-b055-ff26058bdd06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Apr 22 20:09:39.160113 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:39.160089    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="020d2e80-1a3d-4aa3-a220-9c626491fefd" path="/var/lib/kubelet/pods/020d2e80-1a3d-4aa3-a220-9c626491fefd/volumes"
Apr 22 20:09:40.199679 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:40.199644    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv"
Apr 22 20:09:42.848493 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:42.848459    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv"]
Apr 22 20:09:42.848852 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:42.848710    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv" podUID="444625ef-d103-42c3-acaf-b3e82a9d48d0" containerName="model-chainer" containerID="cri-o://e6c5d7087d8bcbcdb8ecc3226fcdadd701da0953db5c0041e329c92f260afe41" gracePeriod=30
Apr 22 20:09:42.993437 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:42.993385    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht"]
Apr 22 20:09:42.993763 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:42.993748    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="020d2e80-1a3d-4aa3-a220-9c626491fefd" containerName="switch-graph-bee44"
Apr 22 20:09:42.993763 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:42.993765    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="020d2e80-1a3d-4aa3-a220-9c626491fefd" containerName="switch-graph-bee44"
Apr 22 20:09:42.993853 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:42.993823    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="020d2e80-1a3d-4aa3-a220-9c626491fefd" containerName="switch-graph-bee44"
Apr 22 20:09:42.997885 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:42.997863    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht"
Apr 22 20:09:43.002522 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:43.002498    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht"]
Apr 22 20:09:43.010262 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:43.010241    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht"
Apr 22 20:09:43.038140 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:43.038114    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d"]
Apr 22 20:09:43.039106 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:43.038473    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d" podUID="2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a" containerName="kserve-container" containerID="cri-o://c9c5f209ab51ca42b609f1982a38e9dfce868621090b24b245db61da4fdd0e43" gracePeriod=30
Apr 22 20:09:43.147952 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:43.147622    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht"]
Apr 22 20:09:43.151303 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:09:43.151271    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65bcf10_d2ce_4479_81d7_adf2f29ff328.slice/crio-c660bcd8e1e2ede3e20c8c0e1d4ec4560e46a39939dcb71c10d04dcd9615fcd9 WatchSource:0}: Error finding container c660bcd8e1e2ede3e20c8c0e1d4ec4560e46a39939dcb71c10d04dcd9615fcd9: Status 404 returned error can't find the container with id c660bcd8e1e2ede3e20c8c0e1d4ec4560e46a39939dcb71c10d04dcd9615fcd9
Apr 22 20:09:43.218115 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:43.218029    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht" event={"ID":"b65bcf10-d2ce-4479-81d7-adf2f29ff328","Type":"ContainerStarted","Data":"c660bcd8e1e2ede3e20c8c0e1d4ec4560e46a39939dcb71c10d04dcd9615fcd9"}
Apr 22 20:09:44.222406 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:44.222366    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht" event={"ID":"b65bcf10-d2ce-4479-81d7-adf2f29ff328","Type":"ContainerStarted","Data":"e6b220af273c5a9017625b7eb100c70643eb3a6dc2ac86b43656784acc74f15f"}
Apr 22 20:09:44.222885 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:44.222612    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht"
Apr 22 20:09:44.223767 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:44.223734    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht" podUID="b65bcf10-d2ce-4479-81d7-adf2f29ff328" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Apr 22 20:09:44.240162 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:44.240122    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht" podStartSLOduration=2.24010924 podStartE2EDuration="2.24010924s" podCreationTimestamp="2026-04-22 20:09:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:09:44.238120951 +0000 UTC m=+735.655784621" watchObservedRunningTime="2026-04-22 20:09:44.24010924 +0000 UTC m=+735.657772884"
Apr 22 20:09:45.157565 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:45.157525    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d" podUID="2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Apr 22 20:09:45.199020 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:45.198987    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv" podUID="444625ef-d103-42c3-acaf-b3e82a9d48d0" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:09:45.225459 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:45.225406    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht" podUID="b65bcf10-d2ce-4479-81d7-adf2f29ff328" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Apr 22 20:09:47.076181 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:47.076159    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d"
Apr 22 20:09:47.179748 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:47.179674    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a-kserve-provision-location\") pod \"2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a\" (UID: \"2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a\") "
Apr 22 20:09:47.179994 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:47.179969    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a" (UID: "2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:09:47.232938 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:47.232907    2574 generic.go:358] "Generic (PLEG): container finished" podID="2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a" containerID="c9c5f209ab51ca42b609f1982a38e9dfce868621090b24b245db61da4fdd0e43" exitCode=0
Apr 22 20:09:47.233089 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:47.232981    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d"
Apr 22 20:09:47.233089 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:47.232997    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d" event={"ID":"2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a","Type":"ContainerDied","Data":"c9c5f209ab51ca42b609f1982a38e9dfce868621090b24b245db61da4fdd0e43"}
Apr 22 20:09:47.233089 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:47.233040    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d" event={"ID":"2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a","Type":"ContainerDied","Data":"9c36ea7f3529a96fe64b6312682c04c987914cbb33c62353d47145a98d0f0ae8"}
Apr 22 20:09:47.233089 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:47.233058    2574 scope.go:117] "RemoveContainer" containerID="c9c5f209ab51ca42b609f1982a38e9dfce868621090b24b245db61da4fdd0e43"
Apr 22 20:09:47.247056 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:47.247034    2574 scope.go:117] "RemoveContainer" containerID="5d09fda83f609e00de6a308e15e539537e2ab3452ef8bb29e67ec90753ccd857"
Apr 22 20:09:47.255174 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:47.255157    2574 scope.go:117] "RemoveContainer" containerID="c9c5f209ab51ca42b609f1982a38e9dfce868621090b24b245db61da4fdd0e43"
Apr 22 20:09:47.255525 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:09:47.255466    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9c5f209ab51ca42b609f1982a38e9dfce868621090b24b245db61da4fdd0e43\": container with ID starting with c9c5f209ab51ca42b609f1982a38e9dfce868621090b24b245db61da4fdd0e43 not found: ID does not exist" containerID="c9c5f209ab51ca42b609f1982a38e9dfce868621090b24b245db61da4fdd0e43"
Apr 22 20:09:47.255625 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:47.255522    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9c5f209ab51ca42b609f1982a38e9dfce868621090b24b245db61da4fdd0e43"} err="failed to get container status \"c9c5f209ab51ca42b609f1982a38e9dfce868621090b24b245db61da4fdd0e43\": rpc error: code = NotFound desc = could not find container \"c9c5f209ab51ca42b609f1982a38e9dfce868621090b24b245db61da4fdd0e43\": container with ID starting with c9c5f209ab51ca42b609f1982a38e9dfce868621090b24b245db61da4fdd0e43 not found: ID does not exist"
Apr 22 20:09:47.255625 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:47.255566    2574 scope.go:117] "RemoveContainer" containerID="5d09fda83f609e00de6a308e15e539537e2ab3452ef8bb29e67ec90753ccd857"
Apr 22 20:09:47.256038 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:09:47.255981    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d09fda83f609e00de6a308e15e539537e2ab3452ef8bb29e67ec90753ccd857\": container with ID starting with 5d09fda83f609e00de6a308e15e539537e2ab3452ef8bb29e67ec90753ccd857 not found: ID does not exist" containerID="5d09fda83f609e00de6a308e15e539537e2ab3452ef8bb29e67ec90753ccd857"
Apr 22 20:09:47.256122 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:47.256047    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d09fda83f609e00de6a308e15e539537e2ab3452ef8bb29e67ec90753ccd857"} err="failed to get container status \"5d09fda83f609e00de6a308e15e539537e2ab3452ef8bb29e67ec90753ccd857\": rpc error: code = NotFound desc = could not find container \"5d09fda83f609e00de6a308e15e539537e2ab3452ef8bb29e67ec90753ccd857\": container with ID starting with 5d09fda83f609e00de6a308e15e539537e2ab3452ef8bb29e67ec90753ccd857 not found: ID does not exist"
Apr 22 20:09:47.257719 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:47.257557    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d"]
Apr 22 20:09:47.259046 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:47.259024    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6984557478-8rb9d"]
Apr 22 20:09:47.280361 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:47.280331    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a-kserve-provision-location\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:09:49.117638 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:49.117601    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt" podUID="f7d922f9-3c12-411c-b055-ff26058bdd06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Apr 22 20:09:49.160002 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:49.159976    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a" path="/var/lib/kubelet/pods/2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a/volumes"
Apr 22 20:09:50.198915 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:50.198874    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv" podUID="444625ef-d103-42c3-acaf-b3e82a9d48d0" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:09:55.198913 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:55.198875    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv" podUID="444625ef-d103-42c3-acaf-b3e82a9d48d0" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:09:55.199263 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:55.198972    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv"
Apr 22 20:09:55.226235 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:55.226206    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht" podUID="b65bcf10-d2ce-4479-81d7-adf2f29ff328" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Apr 22 20:09:59.118120 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:09:59.118095    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt"
Apr 22 20:10:00.198573 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:00.198538    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv" podUID="444625ef-d103-42c3-acaf-b3e82a9d48d0" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:10:05.198289 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:05.198251    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv" podUID="444625ef-d103-42c3-acaf-b3e82a9d48d0" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:10:05.225951 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:05.225915    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht" podUID="b65bcf10-d2ce-4479-81d7-adf2f29ff328" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Apr 22 20:10:10.198647 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:10.198614    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv" podUID="444625ef-d103-42c3-acaf-b3e82a9d48d0" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:10:12.984841 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:12.984820    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv"
Apr 22 20:10:13.059918 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:13.059893    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/444625ef-d103-42c3-acaf-b3e82a9d48d0-proxy-tls\") pod \"444625ef-d103-42c3-acaf-b3e82a9d48d0\" (UID: \"444625ef-d103-42c3-acaf-b3e82a9d48d0\") "
Apr 22 20:10:13.060052 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:13.059950    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/444625ef-d103-42c3-acaf-b3e82a9d48d0-openshift-service-ca-bundle\") pod \"444625ef-d103-42c3-acaf-b3e82a9d48d0\" (UID: \"444625ef-d103-42c3-acaf-b3e82a9d48d0\") "
Apr 22 20:10:13.060281 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:13.060260    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/444625ef-d103-42c3-acaf-b3e82a9d48d0-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "444625ef-d103-42c3-acaf-b3e82a9d48d0" (UID: "444625ef-d103-42c3-acaf-b3e82a9d48d0"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:10:13.061943 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:13.061919    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/444625ef-d103-42c3-acaf-b3e82a9d48d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "444625ef-d103-42c3-acaf-b3e82a9d48d0" (UID: "444625ef-d103-42c3-acaf-b3e82a9d48d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:10:13.160403 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:13.160346    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/444625ef-d103-42c3-acaf-b3e82a9d48d0-proxy-tls\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:10:13.160403 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:13.160364    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/444625ef-d103-42c3-acaf-b3e82a9d48d0-openshift-service-ca-bundle\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:10:13.310709 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:13.310679    2574 generic.go:358] "Generic (PLEG): container finished" podID="444625ef-d103-42c3-acaf-b3e82a9d48d0" containerID="e6c5d7087d8bcbcdb8ecc3226fcdadd701da0953db5c0041e329c92f260afe41" exitCode=0
Apr 22 20:10:13.310856 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:13.310731    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv"
Apr 22 20:10:13.310856 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:13.310758    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv" event={"ID":"444625ef-d103-42c3-acaf-b3e82a9d48d0","Type":"ContainerDied","Data":"e6c5d7087d8bcbcdb8ecc3226fcdadd701da0953db5c0041e329c92f260afe41"}
Apr 22 20:10:13.310856 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:13.310807    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv" event={"ID":"444625ef-d103-42c3-acaf-b3e82a9d48d0","Type":"ContainerDied","Data":"ee13e1e606a49ea7098f69375f4b17670b79353439072daf4b8bcc65de482737"}
Apr 22 20:10:13.310856 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:13.310824    2574 scope.go:117] "RemoveContainer" containerID="e6c5d7087d8bcbcdb8ecc3226fcdadd701da0953db5c0041e329c92f260afe41"
Apr 22 20:10:13.318246 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:13.318228    2574 scope.go:117] "RemoveContainer" containerID="e6c5d7087d8bcbcdb8ecc3226fcdadd701da0953db5c0041e329c92f260afe41"
Apr 22 20:10:13.318505 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:10:13.318484    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6c5d7087d8bcbcdb8ecc3226fcdadd701da0953db5c0041e329c92f260afe41\": container with ID starting with e6c5d7087d8bcbcdb8ecc3226fcdadd701da0953db5c0041e329c92f260afe41 not found: ID does not exist" containerID="e6c5d7087d8bcbcdb8ecc3226fcdadd701da0953db5c0041e329c92f260afe41"
Apr 22 20:10:13.318599 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:13.318516    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6c5d7087d8bcbcdb8ecc3226fcdadd701da0953db5c0041e329c92f260afe41"} err="failed to get container status \"e6c5d7087d8bcbcdb8ecc3226fcdadd701da0953db5c0041e329c92f260afe41\": rpc error: code = NotFound desc = could not find container \"e6c5d7087d8bcbcdb8ecc3226fcdadd701da0953db5c0041e329c92f260afe41\": container with ID starting with e6c5d7087d8bcbcdb8ecc3226fcdadd701da0953db5c0041e329c92f260afe41 not found: ID does not exist"
Apr 22 20:10:13.326153 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:13.326124    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv"]
Apr 22 20:10:13.329217 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:13.329196    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-6f87c58cc9-5tnmv"]
Apr 22 20:10:15.160130 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:15.160098    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="444625ef-d103-42c3-acaf-b3e82a9d48d0" path="/var/lib/kubelet/pods/444625ef-d103-42c3-acaf-b3e82a9d48d0/volumes"
Apr 22 20:10:15.226092 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:15.226063    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht" podUID="b65bcf10-d2ce-4479-81d7-adf2f29ff328" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Apr 22 20:10:17.199355 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.199327    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs"]
Apr 22 20:10:17.199798 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.199642    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="444625ef-d103-42c3-acaf-b3e82a9d48d0" containerName="model-chainer"
Apr 22 20:10:17.199798 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.199658    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="444625ef-d103-42c3-acaf-b3e82a9d48d0" containerName="model-chainer"
Apr 22 20:10:17.199798 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.199673    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a" containerName="storage-initializer"
Apr 22 20:10:17.199798 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.199680    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a" containerName="storage-initializer"
Apr 22 20:10:17.199798 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.199693    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a" containerName="kserve-container"
Apr 22 20:10:17.199798 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.199698    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a" containerName="kserve-container"
Apr 22 20:10:17.199798 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.199743    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="444625ef-d103-42c3-acaf-b3e82a9d48d0" containerName="model-chainer"
Apr 22 20:10:17.199798 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.199755    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="2596b2c6-6621-4c8b-b3ab-e4e4e9a7813a" containerName="kserve-container"
Apr 22 20:10:17.204186 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.204167    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs"
Apr 22 20:10:17.207092 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.206818    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-51a0f-serving-cert\""
Apr 22 20:10:17.207092 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.206851    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 22 20:10:17.207346 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.207107    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-51a0f-kube-rbac-proxy-sar-config\""
Apr 22 20:10:17.208935 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.208907    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs"]
Apr 22 20:10:17.290320 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.290285    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c228916-cb45-4842-ba95-7dcc429cff84-openshift-service-ca-bundle\") pod \"switch-graph-51a0f-79d4985c5c-t5bhs\" (UID: \"4c228916-cb45-4842-ba95-7dcc429cff84\") " pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs"
Apr 22 20:10:17.290507 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.290343    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c228916-cb45-4842-ba95-7dcc429cff84-proxy-tls\") pod \"switch-graph-51a0f-79d4985c5c-t5bhs\" (UID: \"4c228916-cb45-4842-ba95-7dcc429cff84\") " pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs"
Apr 22 20:10:17.391327 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.391295    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c228916-cb45-4842-ba95-7dcc429cff84-openshift-service-ca-bundle\") pod \"switch-graph-51a0f-79d4985c5c-t5bhs\" (UID: \"4c228916-cb45-4842-ba95-7dcc429cff84\") " pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs"
Apr 22 20:10:17.391526 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.391339    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c228916-cb45-4842-ba95-7dcc429cff84-proxy-tls\") pod \"switch-graph-51a0f-79d4985c5c-t5bhs\" (UID: \"4c228916-cb45-4842-ba95-7dcc429cff84\") " pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs"
Apr 22 20:10:17.392033 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.392006    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c228916-cb45-4842-ba95-7dcc429cff84-openshift-service-ca-bundle\") pod \"switch-graph-51a0f-79d4985c5c-t5bhs\" (UID: \"4c228916-cb45-4842-ba95-7dcc429cff84\") " pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs"
Apr 22 20:10:17.393832 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.393814    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c228916-cb45-4842-ba95-7dcc429cff84-proxy-tls\") pod \"switch-graph-51a0f-79d4985c5c-t5bhs\" (UID: \"4c228916-cb45-4842-ba95-7dcc429cff84\") " pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs"
Apr 22 20:10:17.516602 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.516541    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs"
Apr 22 20:10:17.635648 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:17.635617    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs"]
Apr 22 20:10:17.638392 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:10:17.638359    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c228916_cb45_4842_ba95_7dcc429cff84.slice/crio-7a914e153f1a776e3961dd47d1c3910daa7b2c16c6ecc1544af041e7aedcb0fb WatchSource:0}: Error finding container 7a914e153f1a776e3961dd47d1c3910daa7b2c16c6ecc1544af041e7aedcb0fb: Status 404 returned error can't find the container with id 7a914e153f1a776e3961dd47d1c3910daa7b2c16c6ecc1544af041e7aedcb0fb
Apr 22 20:10:18.327989 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:18.327958    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs" event={"ID":"4c228916-cb45-4842-ba95-7dcc429cff84","Type":"ContainerStarted","Data":"05df8f35c099f05d537788e643a2b80c37db4016c73f4da617b09d5a2b1cc9c0"}
Apr 22 20:10:18.327989 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:18.327990    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs" event={"ID":"4c228916-cb45-4842-ba95-7dcc429cff84","Type":"ContainerStarted","Data":"7a914e153f1a776e3961dd47d1c3910daa7b2c16c6ecc1544af041e7aedcb0fb"}
Apr 22 20:10:18.328385 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:18.328080    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs"
Apr 22 20:10:18.343144 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:18.343105    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs" podStartSLOduration=1.343091527 podStartE2EDuration="1.343091527s" podCreationTimestamp="2026-04-22 20:10:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:10:18.341545291 +0000 UTC m=+769.759208930" watchObservedRunningTime="2026-04-22 20:10:18.343091527 +0000 UTC m=+769.760755173"
Apr 22 20:10:24.336994 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:24.336967    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs"
Apr 22 20:10:25.225839 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:25.225801    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht" podUID="b65bcf10-d2ce-4479-81d7-adf2f29ff328" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Apr 22 20:10:35.226640 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:35.226602    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht"
Apr 22 20:10:53.033448 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:53.033348    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4"]
Apr 22 20:10:53.036601 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:53.036582    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4"
Apr 22 20:10:53.039037 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:53.039013    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-cbc99-kube-rbac-proxy-sar-config\""
Apr 22 20:10:53.039037 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:53.039024    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-cbc99-serving-cert\""
Apr 22 20:10:53.044432 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:53.044389    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4"]
Apr 22 20:10:53.146592 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:53.146563    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99ab5b2e-9ec2-4ffd-b8ae-48148dd81967-openshift-service-ca-bundle\") pod \"sequence-graph-cbc99-75c54f6d8c-vj6s4\" (UID: \"99ab5b2e-9ec2-4ffd-b8ae-48148dd81967\") " pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4"
Apr 22 20:10:53.146770 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:53.146611    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99ab5b2e-9ec2-4ffd-b8ae-48148dd81967-proxy-tls\") pod \"sequence-graph-cbc99-75c54f6d8c-vj6s4\" (UID: \"99ab5b2e-9ec2-4ffd-b8ae-48148dd81967\") " pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4"
Apr 22 20:10:53.247141 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:53.247110    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99ab5b2e-9ec2-4ffd-b8ae-48148dd81967-openshift-service-ca-bundle\") pod \"sequence-graph-cbc99-75c54f6d8c-vj6s4\" (UID: \"99ab5b2e-9ec2-4ffd-b8ae-48148dd81967\") " pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4"
Apr 22 20:10:53.247300 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:53.247158    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99ab5b2e-9ec2-4ffd-b8ae-48148dd81967-proxy-tls\") pod \"sequence-graph-cbc99-75c54f6d8c-vj6s4\" (UID: \"99ab5b2e-9ec2-4ffd-b8ae-48148dd81967\") " pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4"
Apr 22 20:10:53.247796 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:53.247771    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99ab5b2e-9ec2-4ffd-b8ae-48148dd81967-openshift-service-ca-bundle\") pod \"sequence-graph-cbc99-75c54f6d8c-vj6s4\" (UID: \"99ab5b2e-9ec2-4ffd-b8ae-48148dd81967\") " pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4"
Apr 22 20:10:53.249822 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:53.249801    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99ab5b2e-9ec2-4ffd-b8ae-48148dd81967-proxy-tls\") pod \"sequence-graph-cbc99-75c54f6d8c-vj6s4\" (UID: \"99ab5b2e-9ec2-4ffd-b8ae-48148dd81967\") " pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4"
Apr 22 20:10:53.347537 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:53.347456    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4"
Apr 22 20:10:53.464119 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:53.464094    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4"]
Apr 22 20:10:53.466231 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:10:53.466206    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ab5b2e_9ec2_4ffd_b8ae_48148dd81967.slice/crio-bd451ba71bc70b9e204241b3bd6b2a9cfa2d37a25921b1b6b0a64b2106d2d379 WatchSource:0}: Error finding container bd451ba71bc70b9e204241b3bd6b2a9cfa2d37a25921b1b6b0a64b2106d2d379: Status 404 returned error can't find the container with id bd451ba71bc70b9e204241b3bd6b2a9cfa2d37a25921b1b6b0a64b2106d2d379
Apr 22 20:10:54.431092 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:54.431060    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4" event={"ID":"99ab5b2e-9ec2-4ffd-b8ae-48148dd81967","Type":"ContainerStarted","Data":"310e1f16090e55a8be8d3671af1b2e3734e6e8dda6c956910a32831d585b8105"}
Apr 22 20:10:54.431092 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:54.431096    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4" event={"ID":"99ab5b2e-9ec2-4ffd-b8ae-48148dd81967","Type":"ContainerStarted","Data":"bd451ba71bc70b9e204241b3bd6b2a9cfa2d37a25921b1b6b0a64b2106d2d379"}
Apr 22 20:10:54.431535 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:54.431204    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4"
Apr 22 20:10:54.445405 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:10:54.445359    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4" podStartSLOduration=1.445345477 podStartE2EDuration="1.445345477s" podCreationTimestamp="2026-04-22 20:10:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:10:54.44473964 +0000 UTC m=+805.862403286" watchObservedRunningTime="2026-04-22 20:10:54.445345477 +0000 UTC m=+805.863009122"
Apr 22 20:11:00.439900 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:11:00.439870    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4"
Apr 22 20:18:31.899362 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:31.899280    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs"]
Apr 22 20:18:31.899878 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:31.899567    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs" podUID="4c228916-cb45-4842-ba95-7dcc429cff84" containerName="switch-graph-51a0f" containerID="cri-o://05df8f35c099f05d537788e643a2b80c37db4016c73f4da617b09d5a2b1cc9c0" gracePeriod=30
Apr 22 20:18:31.993776 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:31.993731    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt"]
Apr 22 20:18:31.994046 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:31.994016    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt" podUID="f7d922f9-3c12-411c-b055-ff26058bdd06" containerName="kserve-container" containerID="cri-o://fb646155cde324315b6d2be3eafe4ce43519bd7e37b5316193ab05fff436c4b7" gracePeriod=30
Apr 22 20:18:32.060095 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:32.060059    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x"]
Apr 22 20:18:32.063882 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:32.063863    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x"
Apr 22 20:18:32.077068 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:32.077040    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x"]
Apr 22 20:18:32.077216 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:32.077198    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x"
Apr 22 20:18:32.210682 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:32.210653    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x"]
Apr 22 20:18:32.213926 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:18:32.213886    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f89ba2_d39c_4674_ae11_8907ac798c79.slice/crio-58201333750d49e1333bad6f3db1888a7ff4206b11de49cf54e3e2e30a3048cd WatchSource:0}: Error finding container 58201333750d49e1333bad6f3db1888a7ff4206b11de49cf54e3e2e30a3048cd: Status 404 returned error can't find the container with id 58201333750d49e1333bad6f3db1888a7ff4206b11de49cf54e3e2e30a3048cd
Apr 22 20:18:32.216177 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:32.216158    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 22 20:18:32.749601 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:32.749565    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x" event={"ID":"63f89ba2-d39c-4674-ae11-8907ac798c79","Type":"ContainerStarted","Data":"45a2d908e8644249eb0ac2af0e6715e0324d1c1a58b44a1022b9efd9f81e336f"}
Apr 22 20:18:32.749601 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:32.749610    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x" event={"ID":"63f89ba2-d39c-4674-ae11-8907ac798c79","Type":"ContainerStarted","Data":"58201333750d49e1333bad6f3db1888a7ff4206b11de49cf54e3e2e30a3048cd"}
Apr 22 20:18:32.749862 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:32.749720    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x"
Apr 22 20:18:32.751104 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:32.751081    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x" podUID="63f89ba2-d39c-4674-ae11-8907ac798c79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Apr 22 20:18:32.764326 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:32.764287    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x" podStartSLOduration=0.764274782 podStartE2EDuration="764.274782ms" podCreationTimestamp="2026-04-22 20:18:32 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:18:32.763148788 +0000 UTC m=+1264.180812434" watchObservedRunningTime="2026-04-22 20:18:32.764274782 +0000 UTC m=+1264.181938428"
Apr 22 20:18:33.753663 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:33.753624    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x" podUID="63f89ba2-d39c-4674-ae11-8907ac798c79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Apr 22 20:18:34.335080 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:34.335045    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs" podUID="4c228916-cb45-4842-ba95-7dcc429cff84" containerName="switch-graph-51a0f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:18:35.132728 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:35.132706    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt"
Apr 22 20:18:35.760588 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:35.760557    2574 generic.go:358] "Generic (PLEG): container finished" podID="f7d922f9-3c12-411c-b055-ff26058bdd06" containerID="fb646155cde324315b6d2be3eafe4ce43519bd7e37b5316193ab05fff436c4b7" exitCode=0
Apr 22 20:18:35.760749 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:35.760606    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt" event={"ID":"f7d922f9-3c12-411c-b055-ff26058bdd06","Type":"ContainerDied","Data":"fb646155cde324315b6d2be3eafe4ce43519bd7e37b5316193ab05fff436c4b7"}
Apr 22 20:18:35.760749 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:35.760628    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt" event={"ID":"f7d922f9-3c12-411c-b055-ff26058bdd06","Type":"ContainerDied","Data":"adf5f6a3a28f2ac2cd79ccabdb59ce5e8ac585c96553c794cbc1422a0bd8d1e9"}
Apr 22 20:18:35.760749 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:35.760628    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt"
Apr 22 20:18:35.760749 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:35.760643    2574 scope.go:117] "RemoveContainer" containerID="fb646155cde324315b6d2be3eafe4ce43519bd7e37b5316193ab05fff436c4b7"
Apr 22 20:18:35.768748 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:35.768731    2574 scope.go:117] "RemoveContainer" containerID="fb646155cde324315b6d2be3eafe4ce43519bd7e37b5316193ab05fff436c4b7"
Apr 22 20:18:35.768998 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:18:35.768982    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb646155cde324315b6d2be3eafe4ce43519bd7e37b5316193ab05fff436c4b7\": container with ID starting with fb646155cde324315b6d2be3eafe4ce43519bd7e37b5316193ab05fff436c4b7 not found: ID does not exist" containerID="fb646155cde324315b6d2be3eafe4ce43519bd7e37b5316193ab05fff436c4b7"
Apr 22 20:18:35.769049 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:35.769006    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb646155cde324315b6d2be3eafe4ce43519bd7e37b5316193ab05fff436c4b7"} err="failed to get container status \"fb646155cde324315b6d2be3eafe4ce43519bd7e37b5316193ab05fff436c4b7\": rpc error: code = NotFound desc = could not find container \"fb646155cde324315b6d2be3eafe4ce43519bd7e37b5316193ab05fff436c4b7\": container with ID starting with fb646155cde324315b6d2be3eafe4ce43519bd7e37b5316193ab05fff436c4b7 not found: ID does not exist"
Apr 22 20:18:35.775350 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:35.775323    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt"]
Apr 22 20:18:35.779197 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:35.779177    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-51a0f-predictor-b84dcfc6d-tt8jt"]
Apr 22 20:18:37.159499 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:37.159463    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7d922f9-3c12-411c-b055-ff26058bdd06" path="/var/lib/kubelet/pods/f7d922f9-3c12-411c-b055-ff26058bdd06/volumes"
Apr 22 20:18:39.335515 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:39.335479    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs" podUID="4c228916-cb45-4842-ba95-7dcc429cff84" containerName="switch-graph-51a0f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:18:43.754640 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:43.754598    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x" podUID="63f89ba2-d39c-4674-ae11-8907ac798c79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Apr 22 20:18:44.335460 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:44.335398    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs" podUID="4c228916-cb45-4842-ba95-7dcc429cff84" containerName="switch-graph-51a0f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:18:44.335654 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:44.335534    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs"
Apr 22 20:18:49.335101 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:49.335054    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs" podUID="4c228916-cb45-4842-ba95-7dcc429cff84" containerName="switch-graph-51a0f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:18:53.753987 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:53.753943    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x" podUID="63f89ba2-d39c-4674-ae11-8907ac798c79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Apr 22 20:18:54.334821 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:54.334771    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs" podUID="4c228916-cb45-4842-ba95-7dcc429cff84" containerName="switch-graph-51a0f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:18:59.334622 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:18:59.334579    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs" podUID="4c228916-cb45-4842-ba95-7dcc429cff84" containerName="switch-graph-51a0f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:19:02.044861 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:02.044828    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs"
Apr 22 20:19:02.181397 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:02.181316    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c228916-cb45-4842-ba95-7dcc429cff84-openshift-service-ca-bundle\") pod \"4c228916-cb45-4842-ba95-7dcc429cff84\" (UID: \"4c228916-cb45-4842-ba95-7dcc429cff84\") "
Apr 22 20:19:02.181397 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:02.181383    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c228916-cb45-4842-ba95-7dcc429cff84-proxy-tls\") pod \"4c228916-cb45-4842-ba95-7dcc429cff84\" (UID: \"4c228916-cb45-4842-ba95-7dcc429cff84\") "
Apr 22 20:19:02.181814 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:02.181781    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c228916-cb45-4842-ba95-7dcc429cff84-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "4c228916-cb45-4842-ba95-7dcc429cff84" (UID: "4c228916-cb45-4842-ba95-7dcc429cff84"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:19:02.183651 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:02.183627    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c228916-cb45-4842-ba95-7dcc429cff84-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4c228916-cb45-4842-ba95-7dcc429cff84" (UID: "4c228916-cb45-4842-ba95-7dcc429cff84"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:19:02.283059 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:02.283011    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c228916-cb45-4842-ba95-7dcc429cff84-openshift-service-ca-bundle\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:19:02.283059 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:02.283055    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c228916-cb45-4842-ba95-7dcc429cff84-proxy-tls\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:19:02.838085 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:02.838050    2574 generic.go:358] "Generic (PLEG): container finished" podID="4c228916-cb45-4842-ba95-7dcc429cff84" containerID="05df8f35c099f05d537788e643a2b80c37db4016c73f4da617b09d5a2b1cc9c0" exitCode=0
Apr 22 20:19:02.838357 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:02.838108    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs"
Apr 22 20:19:02.838357 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:02.838113    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs" event={"ID":"4c228916-cb45-4842-ba95-7dcc429cff84","Type":"ContainerDied","Data":"05df8f35c099f05d537788e643a2b80c37db4016c73f4da617b09d5a2b1cc9c0"}
Apr 22 20:19:02.838357 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:02.838209    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs" event={"ID":"4c228916-cb45-4842-ba95-7dcc429cff84","Type":"ContainerDied","Data":"7a914e153f1a776e3961dd47d1c3910daa7b2c16c6ecc1544af041e7aedcb0fb"}
Apr 22 20:19:02.838357 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:02.838224    2574 scope.go:117] "RemoveContainer" containerID="05df8f35c099f05d537788e643a2b80c37db4016c73f4da617b09d5a2b1cc9c0"
Apr 22 20:19:02.847562 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:02.847540    2574 scope.go:117] "RemoveContainer" containerID="05df8f35c099f05d537788e643a2b80c37db4016c73f4da617b09d5a2b1cc9c0"
Apr 22 20:19:02.847865 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:19:02.847847    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05df8f35c099f05d537788e643a2b80c37db4016c73f4da617b09d5a2b1cc9c0\": container with ID starting with 05df8f35c099f05d537788e643a2b80c37db4016c73f4da617b09d5a2b1cc9c0 not found: ID does not exist" containerID="05df8f35c099f05d537788e643a2b80c37db4016c73f4da617b09d5a2b1cc9c0"
Apr 22 20:19:02.847922 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:02.847874    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05df8f35c099f05d537788e643a2b80c37db4016c73f4da617b09d5a2b1cc9c0"} err="failed to get container status \"05df8f35c099f05d537788e643a2b80c37db4016c73f4da617b09d5a2b1cc9c0\": rpc error: code = NotFound desc = could not find container \"05df8f35c099f05d537788e643a2b80c37db4016c73f4da617b09d5a2b1cc9c0\": container with ID starting with 05df8f35c099f05d537788e643a2b80c37db4016c73f4da617b09d5a2b1cc9c0 not found: ID does not exist"
Apr 22 20:19:02.859088 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:02.859059    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs"]
Apr 22 20:19:02.863800 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:02.863773    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-51a0f-79d4985c5c-t5bhs"]
Apr 22 20:19:03.159897 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:03.159817    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c228916-cb45-4842-ba95-7dcc429cff84" path="/var/lib/kubelet/pods/4c228916-cb45-4842-ba95-7dcc429cff84/volumes"
Apr 22 20:19:03.754180 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:03.754139    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x" podUID="63f89ba2-d39c-4674-ae11-8907ac798c79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Apr 22 20:19:07.678669 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:07.678636    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4"]
Apr 22 20:19:07.679148 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:07.678892    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4" podUID="99ab5b2e-9ec2-4ffd-b8ae-48148dd81967" containerName="sequence-graph-cbc99" containerID="cri-o://310e1f16090e55a8be8d3671af1b2e3734e6e8dda6c956910a32831d585b8105" gracePeriod=30
Apr 22 20:19:07.785608 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:07.785570    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht"]
Apr 22 20:19:07.785888 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:07.785863    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht" podUID="b65bcf10-d2ce-4479-81d7-adf2f29ff328" containerName="kserve-container" containerID="cri-o://e6b220af273c5a9017625b7eb100c70643eb3a6dc2ac86b43656784acc74f15f" gracePeriod=30
Apr 22 20:19:07.797737 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:07.797709    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft"]
Apr 22 20:19:07.798044 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:07.798026    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7d922f9-3c12-411c-b055-ff26058bdd06" containerName="kserve-container"
Apr 22 20:19:07.798044 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:07.798043    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7d922f9-3c12-411c-b055-ff26058bdd06" containerName="kserve-container"
Apr 22 20:19:07.798235 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:07.798051    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c228916-cb45-4842-ba95-7dcc429cff84" containerName="switch-graph-51a0f"
Apr 22 20:19:07.798235 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:07.798056    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c228916-cb45-4842-ba95-7dcc429cff84" containerName="switch-graph-51a0f"
Apr 22 20:19:07.798235 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:07.798117    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="4c228916-cb45-4842-ba95-7dcc429cff84" containerName="switch-graph-51a0f"
Apr 22 20:19:07.798235 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:07.798125    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="f7d922f9-3c12-411c-b055-ff26058bdd06" containerName="kserve-container"
Apr 22 20:19:07.801906 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:07.801888    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft"
Apr 22 20:19:07.807332 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:07.807309    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft"]
Apr 22 20:19:07.812118 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:07.812098    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft"
Apr 22 20:19:07.963134 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:07.963108    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft"]
Apr 22 20:19:07.966244 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:19:07.966216    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a42f24_748d_4ac0_9072_529dbd3780e7.slice/crio-d39e68955c7b036c69b06c50ed0dfa765428eadc4f8642e6731e9a2ebcfa20b9 WatchSource:0}: Error finding container d39e68955c7b036c69b06c50ed0dfa765428eadc4f8642e6731e9a2ebcfa20b9: Status 404 returned error can't find the container with id d39e68955c7b036c69b06c50ed0dfa765428eadc4f8642e6731e9a2ebcfa20b9
Apr 22 20:19:08.862317 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:08.862277    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft" event={"ID":"c6a42f24-748d-4ac0-9072-529dbd3780e7","Type":"ContainerStarted","Data":"e3cb06d06526e5f82e2cb319f8064abaadb8877aa3cb20199bd2659b08fb3425"}
Apr 22 20:19:08.862317 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:08.862323    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft" event={"ID":"c6a42f24-748d-4ac0-9072-529dbd3780e7","Type":"ContainerStarted","Data":"d39e68955c7b036c69b06c50ed0dfa765428eadc4f8642e6731e9a2ebcfa20b9"}
Apr 22 20:19:08.862807 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:08.862483    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft"
Apr 22 20:19:08.863873 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:08.863845    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft" podUID="c6a42f24-748d-4ac0-9072-529dbd3780e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Apr 22 20:19:08.876873 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:08.876835    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft" podStartSLOduration=1.876822846 podStartE2EDuration="1.876822846s" podCreationTimestamp="2026-04-22 20:19:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:19:08.875870843 +0000 UTC m=+1300.293534488" watchObservedRunningTime="2026-04-22 20:19:08.876822846 +0000 UTC m=+1300.294486490"
Apr 22 20:19:09.867589 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:09.867543    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft" podUID="c6a42f24-748d-4ac0-9072-529dbd3780e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Apr 22 20:19:10.438650 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:10.438607    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4" podUID="99ab5b2e-9ec2-4ffd-b8ae-48148dd81967" containerName="sequence-graph-cbc99" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:19:10.876143 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:10.876112    2574 generic.go:358] "Generic (PLEG): container finished" podID="b65bcf10-d2ce-4479-81d7-adf2f29ff328" containerID="e6b220af273c5a9017625b7eb100c70643eb3a6dc2ac86b43656784acc74f15f" exitCode=0
Apr 22 20:19:10.876485 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:10.876192    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht" event={"ID":"b65bcf10-d2ce-4479-81d7-adf2f29ff328","Type":"ContainerDied","Data":"e6b220af273c5a9017625b7eb100c70643eb3a6dc2ac86b43656784acc74f15f"}
Apr 22 20:19:10.928372 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:10.928353    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht"
Apr 22 20:19:11.880925 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:11.880879    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht" event={"ID":"b65bcf10-d2ce-4479-81d7-adf2f29ff328","Type":"ContainerDied","Data":"c660bcd8e1e2ede3e20c8c0e1d4ec4560e46a39939dcb71c10d04dcd9615fcd9"}
Apr 22 20:19:11.883272 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:11.880932    2574 scope.go:117] "RemoveContainer" containerID="e6b220af273c5a9017625b7eb100c70643eb3a6dc2ac86b43656784acc74f15f"
Apr 22 20:19:11.883272 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:11.880932    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht"
Apr 22 20:19:11.896573 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:11.896551    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht"]
Apr 22 20:19:11.899513 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:11.899493    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cbc99-predictor-78d96b46b6-jp2ht"]
Apr 22 20:19:13.160057 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:13.160028    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b65bcf10-d2ce-4479-81d7-adf2f29ff328" path="/var/lib/kubelet/pods/b65bcf10-d2ce-4479-81d7-adf2f29ff328/volumes"
Apr 22 20:19:13.754238 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:13.754189    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x" podUID="63f89ba2-d39c-4674-ae11-8907ac798c79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Apr 22 20:19:15.439110 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:15.439060    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4" podUID="99ab5b2e-9ec2-4ffd-b8ae-48148dd81967" containerName="sequence-graph-cbc99" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:19:19.868475 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:19.868399    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft" podUID="c6a42f24-748d-4ac0-9072-529dbd3780e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Apr 22 20:19:20.438023 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:20.437978    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4" podUID="99ab5b2e-9ec2-4ffd-b8ae-48148dd81967" containerName="sequence-graph-cbc99" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:19:20.438197 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:20.438095    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4"
Apr 22 20:19:23.755308 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:23.755275    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x"
Apr 22 20:19:25.438304 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:25.438265    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4" podUID="99ab5b2e-9ec2-4ffd-b8ae-48148dd81967" containerName="sequence-graph-cbc99" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:19:29.868473 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:29.868408    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft" podUID="c6a42f24-748d-4ac0-9072-529dbd3780e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Apr 22 20:19:30.438182 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:30.438139    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4" podUID="99ab5b2e-9ec2-4ffd-b8ae-48148dd81967" containerName="sequence-graph-cbc99" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:19:35.438012 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:35.437968    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4" podUID="99ab5b2e-9ec2-4ffd-b8ae-48148dd81967" containerName="sequence-graph-cbc99" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:19:37.811233 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:37.811206    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4"
Apr 22 20:19:37.859572 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:37.859538    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99ab5b2e-9ec2-4ffd-b8ae-48148dd81967-openshift-service-ca-bundle\") pod \"99ab5b2e-9ec2-4ffd-b8ae-48148dd81967\" (UID: \"99ab5b2e-9ec2-4ffd-b8ae-48148dd81967\") "
Apr 22 20:19:37.859702 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:37.859629    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99ab5b2e-9ec2-4ffd-b8ae-48148dd81967-proxy-tls\") pod \"99ab5b2e-9ec2-4ffd-b8ae-48148dd81967\" (UID: \"99ab5b2e-9ec2-4ffd-b8ae-48148dd81967\") "
Apr 22 20:19:37.859929 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:37.859907    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ab5b2e-9ec2-4ffd-b8ae-48148dd81967-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "99ab5b2e-9ec2-4ffd-b8ae-48148dd81967" (UID: "99ab5b2e-9ec2-4ffd-b8ae-48148dd81967"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:19:37.861743 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:37.861721    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99ab5b2e-9ec2-4ffd-b8ae-48148dd81967-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "99ab5b2e-9ec2-4ffd-b8ae-48148dd81967" (UID: "99ab5b2e-9ec2-4ffd-b8ae-48148dd81967"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:19:37.955214 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:37.955187    2574 generic.go:358] "Generic (PLEG): container finished" podID="99ab5b2e-9ec2-4ffd-b8ae-48148dd81967" containerID="310e1f16090e55a8be8d3671af1b2e3734e6e8dda6c956910a32831d585b8105" exitCode=0
Apr 22 20:19:37.955353 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:37.955250    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4"
Apr 22 20:19:37.955353 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:37.955274    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4" event={"ID":"99ab5b2e-9ec2-4ffd-b8ae-48148dd81967","Type":"ContainerDied","Data":"310e1f16090e55a8be8d3671af1b2e3734e6e8dda6c956910a32831d585b8105"}
Apr 22 20:19:37.955353 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:37.955312    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4" event={"ID":"99ab5b2e-9ec2-4ffd-b8ae-48148dd81967","Type":"ContainerDied","Data":"bd451ba71bc70b9e204241b3bd6b2a9cfa2d37a25921b1b6b0a64b2106d2d379"}
Apr 22 20:19:37.955353 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:37.955327    2574 scope.go:117] "RemoveContainer" containerID="310e1f16090e55a8be8d3671af1b2e3734e6e8dda6c956910a32831d585b8105"
Apr 22 20:19:37.960202 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:37.960184    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99ab5b2e-9ec2-4ffd-b8ae-48148dd81967-proxy-tls\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:19:37.960292 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:37.960208    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99ab5b2e-9ec2-4ffd-b8ae-48148dd81967-openshift-service-ca-bundle\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:19:37.964964 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:37.964941    2574 scope.go:117] "RemoveContainer" containerID="310e1f16090e55a8be8d3671af1b2e3734e6e8dda6c956910a32831d585b8105"
Apr 22 20:19:37.965206 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:19:37.965187    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"310e1f16090e55a8be8d3671af1b2e3734e6e8dda6c956910a32831d585b8105\": container with ID starting with 310e1f16090e55a8be8d3671af1b2e3734e6e8dda6c956910a32831d585b8105 not found: ID does not exist" containerID="310e1f16090e55a8be8d3671af1b2e3734e6e8dda6c956910a32831d585b8105"
Apr 22 20:19:37.965259 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:37.965214    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"310e1f16090e55a8be8d3671af1b2e3734e6e8dda6c956910a32831d585b8105"} err="failed to get container status \"310e1f16090e55a8be8d3671af1b2e3734e6e8dda6c956910a32831d585b8105\": rpc error: code = NotFound desc = could not find container \"310e1f16090e55a8be8d3671af1b2e3734e6e8dda6c956910a32831d585b8105\": container with ID starting with 310e1f16090e55a8be8d3671af1b2e3734e6e8dda6c956910a32831d585b8105 not found: ID does not exist"
Apr 22 20:19:37.976488 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:37.976462    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4"]
Apr 22 20:19:37.977683 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:37.977664    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-cbc99-75c54f6d8c-vj6s4"]
Apr 22 20:19:39.160972 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:39.160939    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99ab5b2e-9ec2-4ffd-b8ae-48148dd81967" path="/var/lib/kubelet/pods/99ab5b2e-9ec2-4ffd-b8ae-48148dd81967/volumes"
Apr 22 20:19:39.867815 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:39.867772    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft" podUID="c6a42f24-748d-4ac0-9072-529dbd3780e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Apr 22 20:19:42.130928 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:42.130851    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k"]
Apr 22 20:19:42.131254 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:42.131156    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b65bcf10-d2ce-4479-81d7-adf2f29ff328" containerName="kserve-container"
Apr 22 20:19:42.131254 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:42.131167    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b65bcf10-d2ce-4479-81d7-adf2f29ff328" containerName="kserve-container"
Apr 22 20:19:42.131254 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:42.131183    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99ab5b2e-9ec2-4ffd-b8ae-48148dd81967" containerName="sequence-graph-cbc99"
Apr 22 20:19:42.131254 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:42.131190    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="99ab5b2e-9ec2-4ffd-b8ae-48148dd81967" containerName="sequence-graph-cbc99"
Apr 22 20:19:42.131396 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:42.131269    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="b65bcf10-d2ce-4479-81d7-adf2f29ff328" containerName="kserve-container"
Apr 22 20:19:42.131396 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:42.131279    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="99ab5b2e-9ec2-4ffd-b8ae-48148dd81967" containerName="sequence-graph-cbc99"
Apr 22 20:19:42.135319 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:42.135298    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k"
Apr 22 20:19:42.137903 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:42.137877    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-cb62f-kube-rbac-proxy-sar-config\""
Apr 22 20:19:42.138020 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:42.137901    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-cb62f-serving-cert\""
Apr 22 20:19:42.138020 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:42.137971    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 22 20:19:42.144861 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:42.144837    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k"]
Apr 22 20:19:42.188424 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:42.188394    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6-openshift-service-ca-bundle\") pod \"ensemble-graph-cb62f-5d59bc556c-w544k\" (UID: \"c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6\") " pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k"
Apr 22 20:19:42.188574 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:42.188468    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6-proxy-tls\") pod \"ensemble-graph-cb62f-5d59bc556c-w544k\" (UID: \"c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6\") " pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k"
Apr 22 20:19:42.289113 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:42.289081    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6-proxy-tls\") pod \"ensemble-graph-cb62f-5d59bc556c-w544k\" (UID: \"c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6\") " pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k"
Apr 22 20:19:42.289288 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:42.289158    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6-openshift-service-ca-bundle\") pod \"ensemble-graph-cb62f-5d59bc556c-w544k\" (UID: \"c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6\") " pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k"
Apr 22 20:19:42.289288 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:19:42.289248    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-cb62f-serving-cert: secret "ensemble-graph-cb62f-serving-cert" not found
Apr 22 20:19:42.289402 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:19:42.289359    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6-proxy-tls podName:c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6 nodeName:}" failed. No retries permitted until 2026-04-22 20:19:42.789336588 +0000 UTC m=+1334.207000226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6-proxy-tls") pod "ensemble-graph-cb62f-5d59bc556c-w544k" (UID: "c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6") : secret "ensemble-graph-cb62f-serving-cert" not found
Apr 22 20:19:42.289780 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:42.289762    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6-openshift-service-ca-bundle\") pod \"ensemble-graph-cb62f-5d59bc556c-w544k\" (UID: \"c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6\") " pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k"
Apr 22 20:19:42.792664 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:42.792625    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6-proxy-tls\") pod \"ensemble-graph-cb62f-5d59bc556c-w544k\" (UID: \"c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6\") " pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k"
Apr 22 20:19:42.795251 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:42.795219    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6-proxy-tls\") pod \"ensemble-graph-cb62f-5d59bc556c-w544k\" (UID: \"c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6\") " pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k"
Apr 22 20:19:43.050466 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:43.050363    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k"
Apr 22 20:19:43.165805 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:43.165669    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k"]
Apr 22 20:19:43.168473 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:19:43.168449    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a1ae3f_a037_4b0c_bcc9_aa3ea61818d6.slice/crio-f6b46c057380ab12f26b28fb658d2710ad7334e7aa4dfe5a59e695762256a089 WatchSource:0}: Error finding container f6b46c057380ab12f26b28fb658d2710ad7334e7aa4dfe5a59e695762256a089: Status 404 returned error can't find the container with id f6b46c057380ab12f26b28fb658d2710ad7334e7aa4dfe5a59e695762256a089
Apr 22 20:19:43.974950 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:43.974911    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k" event={"ID":"c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6","Type":"ContainerStarted","Data":"d03938ba116143e2633b3087c4630209e7be4ddcddd72872c09a3ac47d260205"}
Apr 22 20:19:43.974950 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:43.974954    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k" event={"ID":"c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6","Type":"ContainerStarted","Data":"f6b46c057380ab12f26b28fb658d2710ad7334e7aa4dfe5a59e695762256a089"}
Apr 22 20:19:43.975163 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:43.975044    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k"
Apr 22 20:19:43.991223 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:43.991183    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k" podStartSLOduration=1.991170664 podStartE2EDuration="1.991170664s" podCreationTimestamp="2026-04-22 20:19:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:19:43.989460934 +0000 UTC m=+1335.407124576" watchObservedRunningTime="2026-04-22 20:19:43.991170664 +0000 UTC m=+1335.408834338"
Apr 22 20:19:49.868270 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:49.868181    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft" podUID="c6a42f24-748d-4ac0-9072-529dbd3780e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Apr 22 20:19:49.983884 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:49.983857    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k"
Apr 22 20:19:52.207198 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:52.207157    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k"]
Apr 22 20:19:52.207647 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:52.207364    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k" podUID="c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6" containerName="ensemble-graph-cb62f" containerID="cri-o://d03938ba116143e2633b3087c4630209e7be4ddcddd72872c09a3ac47d260205" gracePeriod=30
Apr 22 20:19:52.295394 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:52.295362    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x"]
Apr 22 20:19:52.295714 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:52.295668    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x" podUID="63f89ba2-d39c-4674-ae11-8907ac798c79" containerName="kserve-container" containerID="cri-o://45a2d908e8644249eb0ac2af0e6715e0324d1c1a58b44a1022b9efd9f81e336f" gracePeriod=30
Apr 22 20:19:52.331226 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:52.331187    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn"]
Apr 22 20:19:52.334943 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:52.334919    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn"
Apr 22 20:19:52.343032 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:52.342909    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn"]
Apr 22 20:19:52.346369 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:52.346333    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn"
Apr 22 20:19:52.490656 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:52.490631    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn"]
Apr 22 20:19:52.493295 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:19:52.493266    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25473f75_9bd6_4176_9061_77e70f274bf4.slice/crio-8fff10dbf2816b37e891941ae9b82b787e21b5f9e064d8e2998829b191a6b1b5 WatchSource:0}: Error finding container 8fff10dbf2816b37e891941ae9b82b787e21b5f9e064d8e2998829b191a6b1b5: Status 404 returned error can't find the container with id 8fff10dbf2816b37e891941ae9b82b787e21b5f9e064d8e2998829b191a6b1b5
Apr 22 20:19:53.006514 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:53.006482    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn" event={"ID":"25473f75-9bd6-4176-9061-77e70f274bf4","Type":"ContainerStarted","Data":"8a1c4ba24a188e133d4cb6fcdddbc976adeb8fd776a87b59a9c8d03fd550522e"}
Apr 22 20:19:53.006514 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:53.006519    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn" event={"ID":"25473f75-9bd6-4176-9061-77e70f274bf4","Type":"ContainerStarted","Data":"8fff10dbf2816b37e891941ae9b82b787e21b5f9e064d8e2998829b191a6b1b5"}
Apr 22 20:19:53.006724 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:53.006638    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn"
Apr 22 20:19:53.007649 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:53.007625    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn" podUID="25473f75-9bd6-4176-9061-77e70f274bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Apr 22 20:19:53.032685 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:53.032645    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn" podStartSLOduration=1.032632488 podStartE2EDuration="1.032632488s" podCreationTimestamp="2026-04-22 20:19:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:19:53.03099698 +0000 UTC m=+1344.448660664" watchObservedRunningTime="2026-04-22 20:19:53.032632488 +0000 UTC m=+1344.450296133"
Apr 22 20:19:53.754592 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:53.754553    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x" podUID="63f89ba2-d39c-4674-ae11-8907ac798c79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Apr 22 20:19:54.009673 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:54.009582    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn" podUID="25473f75-9bd6-4176-9061-77e70f274bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Apr 22 20:19:54.982222 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:54.982162    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k" podUID="c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6" containerName="ensemble-graph-cb62f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:19:55.247258 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:55.247197    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x"
Apr 22 20:19:56.016798 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:56.016760    2574 generic.go:358] "Generic (PLEG): container finished" podID="63f89ba2-d39c-4674-ae11-8907ac798c79" containerID="45a2d908e8644249eb0ac2af0e6715e0324d1c1a58b44a1022b9efd9f81e336f" exitCode=0
Apr 22 20:19:56.017234 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:56.016827    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x"
Apr 22 20:19:56.017234 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:56.016848    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x" event={"ID":"63f89ba2-d39c-4674-ae11-8907ac798c79","Type":"ContainerDied","Data":"45a2d908e8644249eb0ac2af0e6715e0324d1c1a58b44a1022b9efd9f81e336f"}
Apr 22 20:19:56.017234 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:56.016883    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x" event={"ID":"63f89ba2-d39c-4674-ae11-8907ac798c79","Type":"ContainerDied","Data":"58201333750d49e1333bad6f3db1888a7ff4206b11de49cf54e3e2e30a3048cd"}
Apr 22 20:19:56.017234 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:56.016900    2574 scope.go:117] "RemoveContainer" containerID="45a2d908e8644249eb0ac2af0e6715e0324d1c1a58b44a1022b9efd9f81e336f"
Apr 22 20:19:56.024642 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:56.024614    2574 scope.go:117] "RemoveContainer" containerID="45a2d908e8644249eb0ac2af0e6715e0324d1c1a58b44a1022b9efd9f81e336f"
Apr 22 20:19:56.024907 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:19:56.024888    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a2d908e8644249eb0ac2af0e6715e0324d1c1a58b44a1022b9efd9f81e336f\": container with ID starting with 45a2d908e8644249eb0ac2af0e6715e0324d1c1a58b44a1022b9efd9f81e336f not found: ID does not exist" containerID="45a2d908e8644249eb0ac2af0e6715e0324d1c1a58b44a1022b9efd9f81e336f"
Apr 22 20:19:56.024907 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:56.024913    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a2d908e8644249eb0ac2af0e6715e0324d1c1a58b44a1022b9efd9f81e336f"} err="failed to get container status \"45a2d908e8644249eb0ac2af0e6715e0324d1c1a58b44a1022b9efd9f81e336f\": rpc error: code = NotFound desc = could not find container \"45a2d908e8644249eb0ac2af0e6715e0324d1c1a58b44a1022b9efd9f81e336f\": container with ID starting with 45a2d908e8644249eb0ac2af0e6715e0324d1c1a58b44a1022b9efd9f81e336f not found: ID does not exist"
Apr 22 20:19:56.036276 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:56.036255    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x"]
Apr 22 20:19:56.039930 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:56.039908    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb62f-predictor-64c499d7dd-7kw6x"]
Apr 22 20:19:57.160239 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:57.160208    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63f89ba2-d39c-4674-ae11-8907ac798c79" path="/var/lib/kubelet/pods/63f89ba2-d39c-4674-ae11-8907ac798c79/volumes"
Apr 22 20:19:59.868622 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:59.868593    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft"
Apr 22 20:19:59.982222 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:19:59.982184    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k" podUID="c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6" containerName="ensemble-graph-cb62f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:20:04.010570 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:04.010522    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn" podUID="25473f75-9bd6-4176-9061-77e70f274bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Apr 22 20:20:04.982203 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:04.982158    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k" podUID="c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6" containerName="ensemble-graph-cb62f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:20:04.982371 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:04.982262    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k"
Apr 22 20:20:09.982042 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:09.982005    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k" podUID="c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6" containerName="ensemble-graph-cb62f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:20:14.010325 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:14.010281    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn" podUID="25473f75-9bd6-4176-9061-77e70f274bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Apr 22 20:20:14.982542 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:14.982498    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k" podUID="c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6" containerName="ensemble-graph-cb62f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:20:17.842813 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:17.842771    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98"]
Apr 22 20:20:17.843186 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:17.843112    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63f89ba2-d39c-4674-ae11-8907ac798c79" containerName="kserve-container"
Apr 22 20:20:17.843186 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:17.843125    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="63f89ba2-d39c-4674-ae11-8907ac798c79" containerName="kserve-container"
Apr 22 20:20:17.843254 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:17.843194    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="63f89ba2-d39c-4674-ae11-8907ac798c79" containerName="kserve-container"
Apr 22 20:20:17.847666 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:17.847623    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98"
Apr 22 20:20:17.850229 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:17.850202    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-03046-kube-rbac-proxy-sar-config\""
Apr 22 20:20:17.850388 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:17.850255    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-03046-serving-cert\""
Apr 22 20:20:17.856460 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:17.856433    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98"]
Apr 22 20:20:17.953859 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:17.953823    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1926f286-d595-4cf8-8264-679f2c380c71-proxy-tls\") pod \"sequence-graph-03046-f87dc9c67-zhb98\" (UID: \"1926f286-d595-4cf8-8264-679f2c380c71\") " pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98"
Apr 22 20:20:17.954006 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:17.953877    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1926f286-d595-4cf8-8264-679f2c380c71-openshift-service-ca-bundle\") pod \"sequence-graph-03046-f87dc9c67-zhb98\" (UID: \"1926f286-d595-4cf8-8264-679f2c380c71\") " pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98"
Apr 22 20:20:18.054692 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:18.054667    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1926f286-d595-4cf8-8264-679f2c380c71-proxy-tls\") pod \"sequence-graph-03046-f87dc9c67-zhb98\" (UID: \"1926f286-d595-4cf8-8264-679f2c380c71\") " pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98"
Apr 22 20:20:18.054874 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:18.054715    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1926f286-d595-4cf8-8264-679f2c380c71-openshift-service-ca-bundle\") pod \"sequence-graph-03046-f87dc9c67-zhb98\" (UID: \"1926f286-d595-4cf8-8264-679f2c380c71\") " pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98"
Apr 22 20:20:18.055279 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:18.055259    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1926f286-d595-4cf8-8264-679f2c380c71-openshift-service-ca-bundle\") pod \"sequence-graph-03046-f87dc9c67-zhb98\" (UID: \"1926f286-d595-4cf8-8264-679f2c380c71\") " pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98"
Apr 22 20:20:18.057220 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:18.057196    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1926f286-d595-4cf8-8264-679f2c380c71-proxy-tls\") pod \"sequence-graph-03046-f87dc9c67-zhb98\" (UID: \"1926f286-d595-4cf8-8264-679f2c380c71\") " pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98"
Apr 22 20:20:18.158621 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:18.158550    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98"
Apr 22 20:20:18.274312 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:18.274264    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98"]
Apr 22 20:20:18.278115 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:20:18.278078    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1926f286_d595_4cf8_8264_679f2c380c71.slice/crio-2e57c40c8172443a75e7fc56e7e912237ce27ba4a9353de39e3049cf5865f081 WatchSource:0}: Error finding container 2e57c40c8172443a75e7fc56e7e912237ce27ba4a9353de39e3049cf5865f081: Status 404 returned error can't find the container with id 2e57c40c8172443a75e7fc56e7e912237ce27ba4a9353de39e3049cf5865f081
Apr 22 20:20:19.084164 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:19.084125    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98" event={"ID":"1926f286-d595-4cf8-8264-679f2c380c71","Type":"ContainerStarted","Data":"eb07d47d9f76b167c909e7ecf89f44d3e83e2ce89fc8b5adc8494f46866bdc7f"}
Apr 22 20:20:19.084164 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:19.084168    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98" event={"ID":"1926f286-d595-4cf8-8264-679f2c380c71","Type":"ContainerStarted","Data":"2e57c40c8172443a75e7fc56e7e912237ce27ba4a9353de39e3049cf5865f081"}
Apr 22 20:20:19.084591 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:19.084202    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98"
Apr 22 20:20:19.100311 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:19.100258    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98" podStartSLOduration=2.100241964 podStartE2EDuration="2.100241964s" podCreationTimestamp="2026-04-22 20:20:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:20:19.098277265 +0000 UTC m=+1370.515940908" watchObservedRunningTime="2026-04-22 20:20:19.100241964 +0000 UTC m=+1370.517905610"
Apr 22 20:20:19.982465 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:19.982430    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k" podUID="c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6" containerName="ensemble-graph-cb62f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:20:22.349489 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:22.349464    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k"
Apr 22 20:20:22.490633 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:22.490543    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6-openshift-service-ca-bundle\") pod \"c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6\" (UID: \"c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6\") "
Apr 22 20:20:22.490633 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:22.490614    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6-proxy-tls\") pod \"c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6\" (UID: \"c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6\") "
Apr 22 20:20:22.490928 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:22.490903    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6" (UID: "c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:20:22.492863 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:22.492838    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6" (UID: "c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:20:22.591965 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:22.591937    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6-openshift-service-ca-bundle\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:20:22.591965 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:22.591961    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6-proxy-tls\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:20:23.095713 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:23.095678    2574 generic.go:358] "Generic (PLEG): container finished" podID="c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6" containerID="d03938ba116143e2633b3087c4630209e7be4ddcddd72872c09a3ac47d260205" exitCode=0
Apr 22 20:20:23.095903 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:23.095747    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k"
Apr 22 20:20:23.095903 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:23.095769    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k" event={"ID":"c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6","Type":"ContainerDied","Data":"d03938ba116143e2633b3087c4630209e7be4ddcddd72872c09a3ac47d260205"}
Apr 22 20:20:23.095903 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:23.095812    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k" event={"ID":"c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6","Type":"ContainerDied","Data":"f6b46c057380ab12f26b28fb658d2710ad7334e7aa4dfe5a59e695762256a089"}
Apr 22 20:20:23.095903 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:23.095831    2574 scope.go:117] "RemoveContainer" containerID="d03938ba116143e2633b3087c4630209e7be4ddcddd72872c09a3ac47d260205"
Apr 22 20:20:23.104071 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:23.104056    2574 scope.go:117] "RemoveContainer" containerID="d03938ba116143e2633b3087c4630209e7be4ddcddd72872c09a3ac47d260205"
Apr 22 20:20:23.104309 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:20:23.104289    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03938ba116143e2633b3087c4630209e7be4ddcddd72872c09a3ac47d260205\": container with ID starting with d03938ba116143e2633b3087c4630209e7be4ddcddd72872c09a3ac47d260205 not found: ID does not exist" containerID="d03938ba116143e2633b3087c4630209e7be4ddcddd72872c09a3ac47d260205"
Apr 22 20:20:23.104368 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:23.104317    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03938ba116143e2633b3087c4630209e7be4ddcddd72872c09a3ac47d260205"} err="failed to get container status \"d03938ba116143e2633b3087c4630209e7be4ddcddd72872c09a3ac47d260205\": rpc error: code = NotFound desc = could not find container \"d03938ba116143e2633b3087c4630209e7be4ddcddd72872c09a3ac47d260205\": container with ID starting with d03938ba116143e2633b3087c4630209e7be4ddcddd72872c09a3ac47d260205 not found: ID does not exist"
Apr 22 20:20:23.119171 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:23.117502    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k"]
Apr 22 20:20:23.121909 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:23.121887    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-cb62f-5d59bc556c-w544k"]
Apr 22 20:20:23.160031 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:23.160008    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6" path="/var/lib/kubelet/pods/c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6/volumes"
Apr 22 20:20:24.010655 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:24.010614    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn" podUID="25473f75-9bd6-4176-9061-77e70f274bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Apr 22 20:20:25.092502 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:25.092472    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98"
Apr 22 20:20:27.938237 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:27.938206    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98"]
Apr 22 20:20:27.938717 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:27.938438    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98" podUID="1926f286-d595-4cf8-8264-679f2c380c71" containerName="sequence-graph-03046" containerID="cri-o://eb07d47d9f76b167c909e7ecf89f44d3e83e2ce89fc8b5adc8494f46866bdc7f" gracePeriod=30
Apr 22 20:20:28.021198 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:28.021163    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft"]
Apr 22 20:20:28.021485 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:28.021461    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft" podUID="c6a42f24-748d-4ac0-9072-529dbd3780e7" containerName="kserve-container" containerID="cri-o://e3cb06d06526e5f82e2cb319f8064abaadb8877aa3cb20199bd2659b08fb3425" gracePeriod=30
Apr 22 20:20:28.038403 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:28.038378    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn"]
Apr 22 20:20:28.038698 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:28.038685    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6" containerName="ensemble-graph-cb62f"
Apr 22 20:20:28.038741 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:28.038699    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6" containerName="ensemble-graph-cb62f"
Apr 22 20:20:28.038775 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:28.038755    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="c2a1ae3f-a037-4b0c-bcc9-aa3ea61818d6" containerName="ensemble-graph-cb62f"
Apr 22 20:20:28.041627 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:28.041612    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn"
Apr 22 20:20:28.048147 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:28.048123    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn"]
Apr 22 20:20:28.052928 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:28.052909    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn"
Apr 22 20:20:28.198745 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:28.198720    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn"]
Apr 22 20:20:28.201360 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:20:28.201328    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0b734e_0537_43a7_a2f6_3f8acf109b02.slice/crio-843bee4eb7cdd696fa2d5b2e0f40477de110935d516ff2f18a1aac87caaf1a69 WatchSource:0}: Error finding container 843bee4eb7cdd696fa2d5b2e0f40477de110935d516ff2f18a1aac87caaf1a69: Status 404 returned error can't find the container with id 843bee4eb7cdd696fa2d5b2e0f40477de110935d516ff2f18a1aac87caaf1a69
Apr 22 20:20:29.117155 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:29.117114    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn" event={"ID":"9b0b734e-0537-43a7-a2f6-3f8acf109b02","Type":"ContainerStarted","Data":"2d37708ea1d3e0b455249374f5983254159e64596e8fc9d5b6523736082f1fbd"}
Apr 22 20:20:29.117155 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:29.117156    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn" event={"ID":"9b0b734e-0537-43a7-a2f6-3f8acf109b02","Type":"ContainerStarted","Data":"843bee4eb7cdd696fa2d5b2e0f40477de110935d516ff2f18a1aac87caaf1a69"}
Apr 22 20:20:29.117677 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:29.117379    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn"
Apr 22 20:20:29.118554 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:29.118535    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn" podUID="9b0b734e-0537-43a7-a2f6-3f8acf109b02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Apr 22 20:20:29.131888 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:29.131854    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn" podStartSLOduration=1.131841789 podStartE2EDuration="1.131841789s" podCreationTimestamp="2026-04-22 20:20:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:20:29.130049041 +0000 UTC m=+1380.547712690" watchObservedRunningTime="2026-04-22 20:20:29.131841789 +0000 UTC m=+1380.549505429"
Apr 22 20:20:29.868133 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:29.868084    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft" podUID="c6a42f24-748d-4ac0-9072-529dbd3780e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Apr 22 20:20:30.090883 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:30.090848    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98" podUID="1926f286-d595-4cf8-8264-679f2c380c71" containerName="sequence-graph-03046" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:20:30.119805 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:30.119726    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn" podUID="9b0b734e-0537-43a7-a2f6-3f8acf109b02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Apr 22 20:20:31.271258 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:31.271235    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft"
Apr 22 20:20:32.126244 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:32.126206    2574 generic.go:358] "Generic (PLEG): container finished" podID="c6a42f24-748d-4ac0-9072-529dbd3780e7" containerID="e3cb06d06526e5f82e2cb319f8064abaadb8877aa3cb20199bd2659b08fb3425" exitCode=0
Apr 22 20:20:32.126400 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:32.126272    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft"
Apr 22 20:20:32.126400 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:32.126298    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft" event={"ID":"c6a42f24-748d-4ac0-9072-529dbd3780e7","Type":"ContainerDied","Data":"e3cb06d06526e5f82e2cb319f8064abaadb8877aa3cb20199bd2659b08fb3425"}
Apr 22 20:20:32.126400 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:32.126337    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft" event={"ID":"c6a42f24-748d-4ac0-9072-529dbd3780e7","Type":"ContainerDied","Data":"d39e68955c7b036c69b06c50ed0dfa765428eadc4f8642e6731e9a2ebcfa20b9"}
Apr 22 20:20:32.126400 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:32.126353    2574 scope.go:117] "RemoveContainer" containerID="e3cb06d06526e5f82e2cb319f8064abaadb8877aa3cb20199bd2659b08fb3425"
Apr 22 20:20:32.134502 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:32.134485    2574 scope.go:117] "RemoveContainer" containerID="e3cb06d06526e5f82e2cb319f8064abaadb8877aa3cb20199bd2659b08fb3425"
Apr 22 20:20:32.134750 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:20:32.134733    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3cb06d06526e5f82e2cb319f8064abaadb8877aa3cb20199bd2659b08fb3425\": container with ID starting with e3cb06d06526e5f82e2cb319f8064abaadb8877aa3cb20199bd2659b08fb3425 not found: ID does not exist" containerID="e3cb06d06526e5f82e2cb319f8064abaadb8877aa3cb20199bd2659b08fb3425"
Apr 22 20:20:32.134823 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:32.134755    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3cb06d06526e5f82e2cb319f8064abaadb8877aa3cb20199bd2659b08fb3425"} err="failed to get container status \"e3cb06d06526e5f82e2cb319f8064abaadb8877aa3cb20199bd2659b08fb3425\": rpc error: code = NotFound desc = could not find container \"e3cb06d06526e5f82e2cb319f8064abaadb8877aa3cb20199bd2659b08fb3425\": container with ID starting with e3cb06d06526e5f82e2cb319f8064abaadb8877aa3cb20199bd2659b08fb3425 not found: ID does not exist"
Apr 22 20:20:32.146197 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:32.146168    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft"]
Apr 22 20:20:32.148805 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:32.148776    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-03046-predictor-f5bf7b6d4-xbtft"]
Apr 22 20:20:33.159114 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:33.159080    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6a42f24-748d-4ac0-9072-529dbd3780e7" path="/var/lib/kubelet/pods/c6a42f24-748d-4ac0-9072-529dbd3780e7/volumes"
Apr 22 20:20:34.010511 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:34.010473    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn" podUID="25473f75-9bd6-4176-9061-77e70f274bf4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Apr 22 20:20:35.091227 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:35.091176    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98" podUID="1926f286-d595-4cf8-8264-679f2c380c71" containerName="sequence-graph-03046" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:20:40.090683 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:40.090647    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98" podUID="1926f286-d595-4cf8-8264-679f2c380c71" containerName="sequence-graph-03046" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:20:40.091151 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:40.090752    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98"
Apr 22 20:20:40.120388 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:40.120362    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn" podUID="9b0b734e-0537-43a7-a2f6-3f8acf109b02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Apr 22 20:20:44.010862 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:44.010825    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn"
Apr 22 20:20:45.090879 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:45.090839    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98" podUID="1926f286-d595-4cf8-8264-679f2c380c71" containerName="sequence-graph-03046" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:20:50.091058 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:50.091024    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98" podUID="1926f286-d595-4cf8-8264-679f2c380c71" containerName="sequence-graph-03046" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:20:50.119931 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:50.119891    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn" podUID="9b0b734e-0537-43a7-a2f6-3f8acf109b02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Apr 22 20:20:55.091116 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:55.091083    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98" podUID="1926f286-d595-4cf8-8264-679f2c380c71" containerName="sequence-graph-03046" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:20:57.961218 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:20:57.961186    2574 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1926f286_d595_4cf8_8264_679f2c380c71.slice/crio-conmon-eb07d47d9f76b167c909e7ecf89f44d3e83e2ce89fc8b5adc8494f46866bdc7f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1926f286_d595_4cf8_8264_679f2c380c71.slice/crio-eb07d47d9f76b167c909e7ecf89f44d3e83e2ce89fc8b5adc8494f46866bdc7f.scope\": RecentStats: unable to find data in memory cache]"
Apr 22 20:20:57.961621 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:20:57.961280    2574 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1926f286_d595_4cf8_8264_679f2c380c71.slice/crio-eb07d47d9f76b167c909e7ecf89f44d3e83e2ce89fc8b5adc8494f46866bdc7f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1926f286_d595_4cf8_8264_679f2c380c71.slice/crio-conmon-eb07d47d9f76b167c909e7ecf89f44d3e83e2ce89fc8b5adc8494f46866bdc7f.scope\": RecentStats: unable to find data in memory cache]"
Apr 22 20:20:57.961621 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:20:57.961443    2574 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1926f286_d595_4cf8_8264_679f2c380c71.slice/crio-2e57c40c8172443a75e7fc56e7e912237ce27ba4a9353de39e3049cf5865f081\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1926f286_d595_4cf8_8264_679f2c380c71.slice/crio-eb07d47d9f76b167c909e7ecf89f44d3e83e2ce89fc8b5adc8494f46866bdc7f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1926f286_d595_4cf8_8264_679f2c380c71.slice/crio-conmon-eb07d47d9f76b167c909e7ecf89f44d3e83e2ce89fc8b5adc8494f46866bdc7f.scope\": RecentStats: unable to find data in memory cache]"
Apr 22 20:20:58.079298 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:58.079276    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98"
Apr 22 20:20:58.195961 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:58.195881    2574 generic.go:358] "Generic (PLEG): container finished" podID="1926f286-d595-4cf8-8264-679f2c380c71" containerID="eb07d47d9f76b167c909e7ecf89f44d3e83e2ce89fc8b5adc8494f46866bdc7f" exitCode=0
Apr 22 20:20:58.196115 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:58.195961    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98"
Apr 22 20:20:58.196115 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:58.195964    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98" event={"ID":"1926f286-d595-4cf8-8264-679f2c380c71","Type":"ContainerDied","Data":"eb07d47d9f76b167c909e7ecf89f44d3e83e2ce89fc8b5adc8494f46866bdc7f"}
Apr 22 20:20:58.196115 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:58.195998    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98" event={"ID":"1926f286-d595-4cf8-8264-679f2c380c71","Type":"ContainerDied","Data":"2e57c40c8172443a75e7fc56e7e912237ce27ba4a9353de39e3049cf5865f081"}
Apr 22 20:20:58.196115 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:58.196013    2574 scope.go:117] "RemoveContainer" containerID="eb07d47d9f76b167c909e7ecf89f44d3e83e2ce89fc8b5adc8494f46866bdc7f"
Apr 22 20:20:58.203946 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:58.203927    2574 scope.go:117] "RemoveContainer" containerID="eb07d47d9f76b167c909e7ecf89f44d3e83e2ce89fc8b5adc8494f46866bdc7f"
Apr 22 20:20:58.204173 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:20:58.204156    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb07d47d9f76b167c909e7ecf89f44d3e83e2ce89fc8b5adc8494f46866bdc7f\": container with ID starting with eb07d47d9f76b167c909e7ecf89f44d3e83e2ce89fc8b5adc8494f46866bdc7f not found: ID does not exist" containerID="eb07d47d9f76b167c909e7ecf89f44d3e83e2ce89fc8b5adc8494f46866bdc7f"
Apr 22 20:20:58.204228 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:58.204180    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb07d47d9f76b167c909e7ecf89f44d3e83e2ce89fc8b5adc8494f46866bdc7f"} err="failed to get container status \"eb07d47d9f76b167c909e7ecf89f44d3e83e2ce89fc8b5adc8494f46866bdc7f\": rpc error: code = NotFound desc = could not find container \"eb07d47d9f76b167c909e7ecf89f44d3e83e2ce89fc8b5adc8494f46866bdc7f\": container with ID starting with eb07d47d9f76b167c909e7ecf89f44d3e83e2ce89fc8b5adc8494f46866bdc7f not found: ID does not exist"
Apr 22 20:20:58.261389 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:58.261365    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1926f286-d595-4cf8-8264-679f2c380c71-openshift-service-ca-bundle\") pod \"1926f286-d595-4cf8-8264-679f2c380c71\" (UID: \"1926f286-d595-4cf8-8264-679f2c380c71\") "
Apr 22 20:20:58.261508 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:58.261449    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1926f286-d595-4cf8-8264-679f2c380c71-proxy-tls\") pod \"1926f286-d595-4cf8-8264-679f2c380c71\" (UID: \"1926f286-d595-4cf8-8264-679f2c380c71\") "
Apr 22 20:20:58.261761 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:58.261738    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1926f286-d595-4cf8-8264-679f2c380c71-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "1926f286-d595-4cf8-8264-679f2c380c71" (UID: "1926f286-d595-4cf8-8264-679f2c380c71"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:20:58.263567 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:58.263551    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1926f286-d595-4cf8-8264-679f2c380c71-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1926f286-d595-4cf8-8264-679f2c380c71" (UID: "1926f286-d595-4cf8-8264-679f2c380c71"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:20:58.362085 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:58.362060    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1926f286-d595-4cf8-8264-679f2c380c71-openshift-service-ca-bundle\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:20:58.362085 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:58.362084    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1926f286-d595-4cf8-8264-679f2c380c71-proxy-tls\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:20:58.515977 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:58.515940    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98"]
Apr 22 20:20:58.517329 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:58.517307    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-03046-f87dc9c67-zhb98"]
Apr 22 20:20:59.160120 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:20:59.160084    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1926f286-d595-4cf8-8264-679f2c380c71" path="/var/lib/kubelet/pods/1926f286-d595-4cf8-8264-679f2c380c71/volumes"
Apr 22 20:21:00.120932 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:00.120884    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn" podUID="9b0b734e-0537-43a7-a2f6-3f8acf109b02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Apr 22 20:21:02.417047 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:02.417009    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc"]
Apr 22 20:21:02.417538 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:02.417349    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1926f286-d595-4cf8-8264-679f2c380c71" containerName="sequence-graph-03046"
Apr 22 20:21:02.417538 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:02.417361    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1926f286-d595-4cf8-8264-679f2c380c71" containerName="sequence-graph-03046"
Apr 22 20:21:02.417538 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:02.417372    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a42f24-748d-4ac0-9072-529dbd3780e7" containerName="kserve-container"
Apr 22 20:21:02.417538 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:02.417378    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a42f24-748d-4ac0-9072-529dbd3780e7" containerName="kserve-container"
Apr 22 20:21:02.417538 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:02.417455    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="c6a42f24-748d-4ac0-9072-529dbd3780e7" containerName="kserve-container"
Apr 22 20:21:02.417538 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:02.417464    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="1926f286-d595-4cf8-8264-679f2c380c71" containerName="sequence-graph-03046"
Apr 22 20:21:02.421704 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:02.421683    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc"
Apr 22 20:21:02.424646 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:02.424441    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-08e8b-serving-cert\""
Apr 22 20:21:02.424646 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:02.424470    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-08e8b-kube-rbac-proxy-sar-config\""
Apr 22 20:21:02.424646 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:02.424556    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 22 20:21:02.426771 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:02.426749    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc"]
Apr 22 20:21:02.598263 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:02.598228    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e8fcd78-f93c-4cc1-bfa8-5133765e10e1-openshift-service-ca-bundle\") pod \"ensemble-graph-08e8b-94b5c9c99-khzcc\" (UID: \"0e8fcd78-f93c-4cc1-bfa8-5133765e10e1\") " pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc"
Apr 22 20:21:02.598263 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:02.598273    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e8fcd78-f93c-4cc1-bfa8-5133765e10e1-proxy-tls\") pod \"ensemble-graph-08e8b-94b5c9c99-khzcc\" (UID: \"0e8fcd78-f93c-4cc1-bfa8-5133765e10e1\") " pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc"
Apr 22 20:21:02.699209 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:02.699174    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e8fcd78-f93c-4cc1-bfa8-5133765e10e1-openshift-service-ca-bundle\") pod \"ensemble-graph-08e8b-94b5c9c99-khzcc\" (UID: \"0e8fcd78-f93c-4cc1-bfa8-5133765e10e1\") " pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc"
Apr 22 20:21:02.699371 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:02.699217    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e8fcd78-f93c-4cc1-bfa8-5133765e10e1-proxy-tls\") pod \"ensemble-graph-08e8b-94b5c9c99-khzcc\" (UID: \"0e8fcd78-f93c-4cc1-bfa8-5133765e10e1\") " pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc"
Apr 22 20:21:02.699371 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:21:02.699331    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-08e8b-serving-cert: secret "ensemble-graph-08e8b-serving-cert" not found
Apr 22 20:21:02.699488 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:21:02.699383    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e8fcd78-f93c-4cc1-bfa8-5133765e10e1-proxy-tls podName:0e8fcd78-f93c-4cc1-bfa8-5133765e10e1 nodeName:}" failed. No retries permitted until 2026-04-22 20:21:03.199368128 +0000 UTC m=+1414.617031752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0e8fcd78-f93c-4cc1-bfa8-5133765e10e1-proxy-tls") pod "ensemble-graph-08e8b-94b5c9c99-khzcc" (UID: "0e8fcd78-f93c-4cc1-bfa8-5133765e10e1") : secret "ensemble-graph-08e8b-serving-cert" not found
Apr 22 20:21:02.699856 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:02.699839    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e8fcd78-f93c-4cc1-bfa8-5133765e10e1-openshift-service-ca-bundle\") pod \"ensemble-graph-08e8b-94b5c9c99-khzcc\" (UID: \"0e8fcd78-f93c-4cc1-bfa8-5133765e10e1\") " pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc"
Apr 22 20:21:03.203299 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:03.203268    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e8fcd78-f93c-4cc1-bfa8-5133765e10e1-proxy-tls\") pod \"ensemble-graph-08e8b-94b5c9c99-khzcc\" (UID: \"0e8fcd78-f93c-4cc1-bfa8-5133765e10e1\") " pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc"
Apr 22 20:21:03.205910 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:03.205891    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e8fcd78-f93c-4cc1-bfa8-5133765e10e1-proxy-tls\") pod \"ensemble-graph-08e8b-94b5c9c99-khzcc\" (UID: \"0e8fcd78-f93c-4cc1-bfa8-5133765e10e1\") " pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc"
Apr 22 20:21:03.335015 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:03.334980    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc"
Apr 22 20:21:03.454517 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:03.454446    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc"]
Apr 22 20:21:03.459031 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:21:03.459000    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8fcd78_f93c_4cc1_bfa8_5133765e10e1.slice/crio-a14e5d3d5f9340a8f604851dba5fc0f3a9076a75ba19d6dfd43e8cb9f3c0b9e1 WatchSource:0}: Error finding container a14e5d3d5f9340a8f604851dba5fc0f3a9076a75ba19d6dfd43e8cb9f3c0b9e1: Status 404 returned error can't find the container with id a14e5d3d5f9340a8f604851dba5fc0f3a9076a75ba19d6dfd43e8cb9f3c0b9e1
Apr 22 20:21:04.214473 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:04.214429    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc" event={"ID":"0e8fcd78-f93c-4cc1-bfa8-5133765e10e1","Type":"ContainerStarted","Data":"336c62935a237286a86f1b8466ed095adb346099aa0269e3020d336d1e227b21"}
Apr 22 20:21:04.214659 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:04.214481    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc" event={"ID":"0e8fcd78-f93c-4cc1-bfa8-5133765e10e1","Type":"ContainerStarted","Data":"a14e5d3d5f9340a8f604851dba5fc0f3a9076a75ba19d6dfd43e8cb9f3c0b9e1"}
Apr 22 20:21:04.228854 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:04.228806    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc" podStartSLOduration=2.228788677 podStartE2EDuration="2.228788677s" podCreationTimestamp="2026-04-22 20:21:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:21:04.22822414 +0000 UTC m=+1415.645887785" watchObservedRunningTime="2026-04-22 20:21:04.228788677 +0000 UTC m=+1415.646452324"
Apr 22 20:21:05.216939 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:05.216907    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc"
Apr 22 20:21:10.120541 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:10.120501    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn" podUID="9b0b734e-0537-43a7-a2f6-3f8acf109b02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Apr 22 20:21:11.226111 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:11.226086    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc"
Apr 22 20:21:20.121465 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:20.121368    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn"
Apr 22 20:21:38.140235 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:38.140199    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv"]
Apr 22 20:21:38.145157 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:38.145139    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv"
Apr 22 20:21:38.147617 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:38.147595    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-b78d5-kube-rbac-proxy-sar-config\""
Apr 22 20:21:38.147748 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:38.147649    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-b78d5-serving-cert\""
Apr 22 20:21:38.151832 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:38.151805    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv"]
Apr 22 20:21:38.262387 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:38.262351    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abeff13e-d81d-45be-9051-225f25903669-openshift-service-ca-bundle\") pod \"sequence-graph-b78d5-6dcd78c574-pfvzv\" (UID: \"abeff13e-d81d-45be-9051-225f25903669\") " pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv"
Apr 22 20:21:38.262387 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:38.262392    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abeff13e-d81d-45be-9051-225f25903669-proxy-tls\") pod \"sequence-graph-b78d5-6dcd78c574-pfvzv\" (UID: \"abeff13e-d81d-45be-9051-225f25903669\") " pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv"
Apr 22 20:21:38.362997 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:38.362961    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abeff13e-d81d-45be-9051-225f25903669-openshift-service-ca-bundle\") pod \"sequence-graph-b78d5-6dcd78c574-pfvzv\" (UID: \"abeff13e-d81d-45be-9051-225f25903669\") " pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv"
Apr 22 20:21:38.362997 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:38.363003    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abeff13e-d81d-45be-9051-225f25903669-proxy-tls\") pod \"sequence-graph-b78d5-6dcd78c574-pfvzv\" (UID: \"abeff13e-d81d-45be-9051-225f25903669\") " pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv"
Apr 22 20:21:38.363243 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:21:38.363105    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-b78d5-serving-cert: secret "sequence-graph-b78d5-serving-cert" not found
Apr 22 20:21:38.363243 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:21:38.363161    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abeff13e-d81d-45be-9051-225f25903669-proxy-tls podName:abeff13e-d81d-45be-9051-225f25903669 nodeName:}" failed. No retries permitted until 2026-04-22 20:21:38.863146294 +0000 UTC m=+1450.280809919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/abeff13e-d81d-45be-9051-225f25903669-proxy-tls") pod "sequence-graph-b78d5-6dcd78c574-pfvzv" (UID: "abeff13e-d81d-45be-9051-225f25903669") : secret "sequence-graph-b78d5-serving-cert" not found
Apr 22 20:21:38.363726 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:38.363702    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abeff13e-d81d-45be-9051-225f25903669-openshift-service-ca-bundle\") pod \"sequence-graph-b78d5-6dcd78c574-pfvzv\" (UID: \"abeff13e-d81d-45be-9051-225f25903669\") " pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv"
Apr 22 20:21:38.866973 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:38.866920    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abeff13e-d81d-45be-9051-225f25903669-proxy-tls\") pod \"sequence-graph-b78d5-6dcd78c574-pfvzv\" (UID: \"abeff13e-d81d-45be-9051-225f25903669\") " pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv"
Apr 22 20:21:38.869478 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:38.869457    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abeff13e-d81d-45be-9051-225f25903669-proxy-tls\") pod \"sequence-graph-b78d5-6dcd78c574-pfvzv\" (UID: \"abeff13e-d81d-45be-9051-225f25903669\") " pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv"
Apr 22 20:21:39.057068 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:39.057024    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv"
Apr 22 20:21:39.175155 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:39.175113    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv"]
Apr 22 20:21:39.177310 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:21:39.177282    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabeff13e_d81d_45be_9051_225f25903669.slice/crio-98c93c749f976c267966b582dac1968bad7d3cef3e95492b7575d19d7a6e8270 WatchSource:0}: Error finding container 98c93c749f976c267966b582dac1968bad7d3cef3e95492b7575d19d7a6e8270: Status 404 returned error can't find the container with id 98c93c749f976c267966b582dac1968bad7d3cef3e95492b7575d19d7a6e8270
Apr 22 20:21:39.314926 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:39.314887    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv" event={"ID":"abeff13e-d81d-45be-9051-225f25903669","Type":"ContainerStarted","Data":"e25cb8276cfa9b0df4b1f8165c3e775a5e882f88991dc6f0a933d5361b2ff2c3"}
Apr 22 20:21:39.314926 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:39.314934    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv" event={"ID":"abeff13e-d81d-45be-9051-225f25903669","Type":"ContainerStarted","Data":"98c93c749f976c267966b582dac1968bad7d3cef3e95492b7575d19d7a6e8270"}
Apr 22 20:21:39.315174 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:39.314987    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv"
Apr 22 20:21:39.330815 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:39.330763    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv" podStartSLOduration=1.330750755 podStartE2EDuration="1.330750755s" podCreationTimestamp="2026-04-22 20:21:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:21:39.328586849 +0000 UTC m=+1450.746250495" watchObservedRunningTime="2026-04-22 20:21:39.330750755 +0000 UTC m=+1450.748414400"
Apr 22 20:21:45.322676 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:21:45.322650    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv"
Apr 22 20:29:17.118547 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:17.118508    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc"]
Apr 22 20:29:17.119102 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:17.118819    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc" podUID="0e8fcd78-f93c-4cc1-bfa8-5133765e10e1" containerName="ensemble-graph-08e8b" containerID="cri-o://336c62935a237286a86f1b8466ed095adb346099aa0269e3020d336d1e227b21" gracePeriod=30
Apr 22 20:29:17.212882 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:17.212847    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn"]
Apr 22 20:29:17.286729 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:17.286695    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq"]
Apr 22 20:29:17.290639 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:17.290617    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq"
Apr 22 20:29:17.304072 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:17.304002    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq"
Apr 22 20:29:17.313377 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:17.306812    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq"]
Apr 22 20:29:17.439257 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:17.439234    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq"]
Apr 22 20:29:17.441894 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:29:17.441864    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3a80d3_4c53_4eaf_b460_9bc232d9f5ef.slice/crio-2f1001f62b77a4144c518730f1ac94f510e033232a4d7c885e5b515be1457b8d WatchSource:0}: Error finding container 2f1001f62b77a4144c518730f1ac94f510e033232a4d7c885e5b515be1457b8d: Status 404 returned error can't find the container with id 2f1001f62b77a4144c518730f1ac94f510e033232a4d7c885e5b515be1457b8d
Apr 22 20:29:17.443621 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:17.443600    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 22 20:29:17.599167 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:17.599133    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq" event={"ID":"dc3a80d3-4c53-4eaf-b460-9bc232d9f5ef","Type":"ContainerStarted","Data":"9ba2cad11a6e80905a193ad2620a8784828a94322fab56687a306dd4adebea5a"}
Apr 22 20:29:17.599286 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:17.599176    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq" event={"ID":"dc3a80d3-4c53-4eaf-b460-9bc232d9f5ef","Type":"ContainerStarted","Data":"2f1001f62b77a4144c518730f1ac94f510e033232a4d7c885e5b515be1457b8d"}
Apr 22 20:29:17.599286 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:17.599245    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn" podUID="25473f75-9bd6-4176-9061-77e70f274bf4" containerName="kserve-container" containerID="cri-o://8a1c4ba24a188e133d4cb6fcdddbc976adeb8fd776a87b59a9c8d03fd550522e" gracePeriod=30
Apr 22 20:29:17.618329 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:17.618282    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq" podStartSLOduration=0.618270343 podStartE2EDuration="618.270343ms" podCreationTimestamp="2026-04-22 20:29:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:29:17.616755961 +0000 UTC m=+1909.034419608" watchObservedRunningTime="2026-04-22 20:29:17.618270343 +0000 UTC m=+1909.035933989"
Apr 22 20:29:18.601834 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:18.601800    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq"
Apr 22 20:29:18.603171 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:18.603141    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq" podUID="dc3a80d3-4c53-4eaf-b460-9bc232d9f5ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Apr 22 20:29:19.605071 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:19.605035    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq" podUID="dc3a80d3-4c53-4eaf-b460-9bc232d9f5ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Apr 22 20:29:20.439927 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:20.439905    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn"
Apr 22 20:29:20.608405 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:20.608320    2574 generic.go:358] "Generic (PLEG): container finished" podID="25473f75-9bd6-4176-9061-77e70f274bf4" containerID="8a1c4ba24a188e133d4cb6fcdddbc976adeb8fd776a87b59a9c8d03fd550522e" exitCode=0
Apr 22 20:29:20.608405 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:20.608380    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn"
Apr 22 20:29:20.608857 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:20.608403    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn" event={"ID":"25473f75-9bd6-4176-9061-77e70f274bf4","Type":"ContainerDied","Data":"8a1c4ba24a188e133d4cb6fcdddbc976adeb8fd776a87b59a9c8d03fd550522e"}
Apr 22 20:29:20.608857 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:20.608463    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn" event={"ID":"25473f75-9bd6-4176-9061-77e70f274bf4","Type":"ContainerDied","Data":"8fff10dbf2816b37e891941ae9b82b787e21b5f9e064d8e2998829b191a6b1b5"}
Apr 22 20:29:20.608857 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:20.608484    2574 scope.go:117] "RemoveContainer" containerID="8a1c4ba24a188e133d4cb6fcdddbc976adeb8fd776a87b59a9c8d03fd550522e"
Apr 22 20:29:20.616583 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:20.616568    2574 scope.go:117] "RemoveContainer" containerID="8a1c4ba24a188e133d4cb6fcdddbc976adeb8fd776a87b59a9c8d03fd550522e"
Apr 22 20:29:20.616843 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:29:20.616822    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a1c4ba24a188e133d4cb6fcdddbc976adeb8fd776a87b59a9c8d03fd550522e\": container with ID starting with 8a1c4ba24a188e133d4cb6fcdddbc976adeb8fd776a87b59a9c8d03fd550522e not found: ID does not exist" containerID="8a1c4ba24a188e133d4cb6fcdddbc976adeb8fd776a87b59a9c8d03fd550522e"
Apr 22 20:29:20.616900 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:20.616851    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a1c4ba24a188e133d4cb6fcdddbc976adeb8fd776a87b59a9c8d03fd550522e"} err="failed to get container status \"8a1c4ba24a188e133d4cb6fcdddbc976adeb8fd776a87b59a9c8d03fd550522e\": rpc error: code = NotFound desc = could not find container \"8a1c4ba24a188e133d4cb6fcdddbc976adeb8fd776a87b59a9c8d03fd550522e\": container with ID starting with 8a1c4ba24a188e133d4cb6fcdddbc976adeb8fd776a87b59a9c8d03fd550522e not found: ID does not exist"
Apr 22 20:29:20.628028 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:20.628005    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn"]
Apr 22 20:29:20.630849 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:20.630829    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-08e8b-predictor-5f949d5976-g29nn"]
Apr 22 20:29:21.160650 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:21.160615    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25473f75-9bd6-4176-9061-77e70f274bf4" path="/var/lib/kubelet/pods/25473f75-9bd6-4176-9061-77e70f274bf4/volumes"
Apr 22 20:29:21.224251 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:21.224215    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc" podUID="0e8fcd78-f93c-4cc1-bfa8-5133765e10e1" containerName="ensemble-graph-08e8b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:29:26.224013 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:26.223970    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc" podUID="0e8fcd78-f93c-4cc1-bfa8-5133765e10e1" containerName="ensemble-graph-08e8b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:29:29.606167 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:29.606076    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq" podUID="dc3a80d3-4c53-4eaf-b460-9bc232d9f5ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Apr 22 20:29:31.223728 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:31.223691    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc" podUID="0e8fcd78-f93c-4cc1-bfa8-5133765e10e1" containerName="ensemble-graph-08e8b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:29:31.224166 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:31.223797    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc"
Apr 22 20:29:36.224150 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:36.224116    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc" podUID="0e8fcd78-f93c-4cc1-bfa8-5133765e10e1" containerName="ensemble-graph-08e8b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:29:39.605735 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:39.605686    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq" podUID="dc3a80d3-4c53-4eaf-b460-9bc232d9f5ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Apr 22 20:29:41.224202 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:41.224164    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc" podUID="0e8fcd78-f93c-4cc1-bfa8-5133765e10e1" containerName="ensemble-graph-08e8b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:29:46.224381 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:46.224336    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc" podUID="0e8fcd78-f93c-4cc1-bfa8-5133765e10e1" containerName="ensemble-graph-08e8b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:29:47.257342 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:47.257315    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc"
Apr 22 20:29:47.311351 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:47.311313    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e8fcd78-f93c-4cc1-bfa8-5133765e10e1-proxy-tls\") pod \"0e8fcd78-f93c-4cc1-bfa8-5133765e10e1\" (UID: \"0e8fcd78-f93c-4cc1-bfa8-5133765e10e1\") "
Apr 22 20:29:47.311351 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:47.311357    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e8fcd78-f93c-4cc1-bfa8-5133765e10e1-openshift-service-ca-bundle\") pod \"0e8fcd78-f93c-4cc1-bfa8-5133765e10e1\" (UID: \"0e8fcd78-f93c-4cc1-bfa8-5133765e10e1\") "
Apr 22 20:29:47.311805 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:47.311774    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8fcd78-f93c-4cc1-bfa8-5133765e10e1-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0e8fcd78-f93c-4cc1-bfa8-5133765e10e1" (UID: "0e8fcd78-f93c-4cc1-bfa8-5133765e10e1"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:29:47.313737 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:47.313705    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e8fcd78-f93c-4cc1-bfa8-5133765e10e1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e8fcd78-f93c-4cc1-bfa8-5133765e10e1" (UID: "0e8fcd78-f93c-4cc1-bfa8-5133765e10e1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:29:47.412195 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:47.412104    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e8fcd78-f93c-4cc1-bfa8-5133765e10e1-proxy-tls\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:29:47.412195 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:47.412135    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e8fcd78-f93c-4cc1-bfa8-5133765e10e1-openshift-service-ca-bundle\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:29:47.686510 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:47.686403    2574 generic.go:358] "Generic (PLEG): container finished" podID="0e8fcd78-f93c-4cc1-bfa8-5133765e10e1" containerID="336c62935a237286a86f1b8466ed095adb346099aa0269e3020d336d1e227b21" exitCode=0
Apr 22 20:29:47.686510 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:47.686502    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc"
Apr 22 20:29:47.686716 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:47.686510    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc" event={"ID":"0e8fcd78-f93c-4cc1-bfa8-5133765e10e1","Type":"ContainerDied","Data":"336c62935a237286a86f1b8466ed095adb346099aa0269e3020d336d1e227b21"}
Apr 22 20:29:47.686716 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:47.686550    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc" event={"ID":"0e8fcd78-f93c-4cc1-bfa8-5133765e10e1","Type":"ContainerDied","Data":"a14e5d3d5f9340a8f604851dba5fc0f3a9076a75ba19d6dfd43e8cb9f3c0b9e1"}
Apr 22 20:29:47.686716 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:47.686566    2574 scope.go:117] "RemoveContainer" containerID="336c62935a237286a86f1b8466ed095adb346099aa0269e3020d336d1e227b21"
Apr 22 20:29:47.695374 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:47.695352    2574 scope.go:117] "RemoveContainer" containerID="336c62935a237286a86f1b8466ed095adb346099aa0269e3020d336d1e227b21"
Apr 22 20:29:47.695677 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:29:47.695647    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"336c62935a237286a86f1b8466ed095adb346099aa0269e3020d336d1e227b21\": container with ID starting with 336c62935a237286a86f1b8466ed095adb346099aa0269e3020d336d1e227b21 not found: ID does not exist" containerID="336c62935a237286a86f1b8466ed095adb346099aa0269e3020d336d1e227b21"
Apr 22 20:29:47.695776 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:47.695687    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"336c62935a237286a86f1b8466ed095adb346099aa0269e3020d336d1e227b21"} err="failed to get container status \"336c62935a237286a86f1b8466ed095adb346099aa0269e3020d336d1e227b21\": rpc error: code = NotFound desc = could not find container \"336c62935a237286a86f1b8466ed095adb346099aa0269e3020d336d1e227b21\": container with ID starting with 336c62935a237286a86f1b8466ed095adb346099aa0269e3020d336d1e227b21 not found: ID does not exist"
Apr 22 20:29:47.706750 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:47.706720    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc"]
Apr 22 20:29:47.710260 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:47.710235    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-08e8b-94b5c9c99-khzcc"]
Apr 22 20:29:49.160656 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:49.160622    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e8fcd78-f93c-4cc1-bfa8-5133765e10e1" path="/var/lib/kubelet/pods/0e8fcd78-f93c-4cc1-bfa8-5133765e10e1/volumes"
Apr 22 20:29:49.605775 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:49.605729    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq" podUID="dc3a80d3-4c53-4eaf-b460-9bc232d9f5ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Apr 22 20:29:52.773139 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:52.773105    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv"]
Apr 22 20:29:52.773566 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:52.773348    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv" podUID="abeff13e-d81d-45be-9051-225f25903669" containerName="sequence-graph-b78d5" containerID="cri-o://e25cb8276cfa9b0df4b1f8165c3e775a5e882f88991dc6f0a933d5361b2ff2c3" gracePeriod=30
Apr 22 20:29:52.866137 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:52.866099    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn"]
Apr 22 20:29:52.866460 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:52.866404    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn" podUID="9b0b734e-0537-43a7-a2f6-3f8acf109b02" containerName="kserve-container" containerID="cri-o://2d37708ea1d3e0b455249374f5983254159e64596e8fc9d5b6523736082f1fbd" gracePeriod=30
Apr 22 20:29:52.893469 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:52.893409    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9"]
Apr 22 20:29:52.893771 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:52.893756    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e8fcd78-f93c-4cc1-bfa8-5133765e10e1" containerName="ensemble-graph-08e8b"
Apr 22 20:29:52.893771 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:52.893771    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e8fcd78-f93c-4cc1-bfa8-5133765e10e1" containerName="ensemble-graph-08e8b"
Apr 22 20:29:52.893852 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:52.893792    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25473f75-9bd6-4176-9061-77e70f274bf4" containerName="kserve-container"
Apr 22 20:29:52.893852 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:52.893798    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="25473f75-9bd6-4176-9061-77e70f274bf4" containerName="kserve-container"
Apr 22 20:29:52.893852 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:52.893847    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="0e8fcd78-f93c-4cc1-bfa8-5133765e10e1" containerName="ensemble-graph-08e8b"
Apr 22 20:29:52.893957 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:52.893857    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="25473f75-9bd6-4176-9061-77e70f274bf4" containerName="kserve-container"
Apr 22 20:29:52.898238 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:52.898213    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9"
Apr 22 20:29:52.908282 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:52.908258    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9"
Apr 22 20:29:52.909787 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:52.909635    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9"]
Apr 22 20:29:53.055428 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:53.055371    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9"]
Apr 22 20:29:53.058753 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:29:53.058717    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcffff8c4_5712_4f2f_ae2a_a930738658a8.slice/crio-fd3b5117ec4bca9fe95a3d6c0c93d4f9596572c2bb3965e0458227289ffe2584 WatchSource:0}: Error finding container fd3b5117ec4bca9fe95a3d6c0c93d4f9596572c2bb3965e0458227289ffe2584: Status 404 returned error can't find the container with id fd3b5117ec4bca9fe95a3d6c0c93d4f9596572c2bb3965e0458227289ffe2584
Apr 22 20:29:53.705509 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:53.705473    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9" event={"ID":"cffff8c4-5712-4f2f-ae2a-a930738658a8","Type":"ContainerStarted","Data":"2ae8cc2fd110eb363ef00eb480b5e9f5800a67419a1699e0c710a9bc97fc4189"}
Apr 22 20:29:53.705509 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:53.705515    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9" event={"ID":"cffff8c4-5712-4f2f-ae2a-a930738658a8","Type":"ContainerStarted","Data":"fd3b5117ec4bca9fe95a3d6c0c93d4f9596572c2bb3965e0458227289ffe2584"}
Apr 22 20:29:53.705725 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:53.705651    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9"
Apr 22 20:29:53.706979 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:53.706951    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9" podUID="cffff8c4-5712-4f2f-ae2a-a930738658a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Apr 22 20:29:53.720256 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:53.720212    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9" podStartSLOduration=1.72019737 podStartE2EDuration="1.72019737s" podCreationTimestamp="2026-04-22 20:29:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:29:53.71928733 +0000 UTC m=+1945.136950977" watchObservedRunningTime="2026-04-22 20:29:53.72019737 +0000 UTC m=+1945.137861015"
Apr 22 20:29:54.709328 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:54.709279    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9" podUID="cffff8c4-5712-4f2f-ae2a-a930738658a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Apr 22 20:29:55.321700 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:55.321659    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv" podUID="abeff13e-d81d-45be-9051-225f25903669" containerName="sequence-graph-b78d5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:29:56.213150 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:56.213122    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn"
Apr 22 20:29:56.718088 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:56.718053    2574 generic.go:358] "Generic (PLEG): container finished" podID="9b0b734e-0537-43a7-a2f6-3f8acf109b02" containerID="2d37708ea1d3e0b455249374f5983254159e64596e8fc9d5b6523736082f1fbd" exitCode=0
Apr 22 20:29:56.718261 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:56.718121    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn"
Apr 22 20:29:56.718261 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:56.718115    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn" event={"ID":"9b0b734e-0537-43a7-a2f6-3f8acf109b02","Type":"ContainerDied","Data":"2d37708ea1d3e0b455249374f5983254159e64596e8fc9d5b6523736082f1fbd"}
Apr 22 20:29:56.718261 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:56.718223    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn" event={"ID":"9b0b734e-0537-43a7-a2f6-3f8acf109b02","Type":"ContainerDied","Data":"843bee4eb7cdd696fa2d5b2e0f40477de110935d516ff2f18a1aac87caaf1a69"}
Apr 22 20:29:56.718261 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:56.718240    2574 scope.go:117] "RemoveContainer" containerID="2d37708ea1d3e0b455249374f5983254159e64596e8fc9d5b6523736082f1fbd"
Apr 22 20:29:56.726727 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:56.726707    2574 scope.go:117] "RemoveContainer" containerID="2d37708ea1d3e0b455249374f5983254159e64596e8fc9d5b6523736082f1fbd"
Apr 22 20:29:56.727012 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:29:56.726990    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d37708ea1d3e0b455249374f5983254159e64596e8fc9d5b6523736082f1fbd\": container with ID starting with 2d37708ea1d3e0b455249374f5983254159e64596e8fc9d5b6523736082f1fbd not found: ID does not exist" containerID="2d37708ea1d3e0b455249374f5983254159e64596e8fc9d5b6523736082f1fbd"
Apr 22 20:29:56.727064 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:56.727023    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d37708ea1d3e0b455249374f5983254159e64596e8fc9d5b6523736082f1fbd"} err="failed to get container status \"2d37708ea1d3e0b455249374f5983254159e64596e8fc9d5b6523736082f1fbd\": rpc error: code = NotFound desc = could not find container \"2d37708ea1d3e0b455249374f5983254159e64596e8fc9d5b6523736082f1fbd\": container with ID starting with 2d37708ea1d3e0b455249374f5983254159e64596e8fc9d5b6523736082f1fbd not found: ID does not exist"
Apr 22 20:29:56.738924 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:56.738889    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn"]
Apr 22 20:29:56.741970 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:56.741943    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b78d5-predictor-5c4d5678c8-k59rn"]
Apr 22 20:29:57.160396 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:57.160313    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b0b734e-0537-43a7-a2f6-3f8acf109b02" path="/var/lib/kubelet/pods/9b0b734e-0537-43a7-a2f6-3f8acf109b02/volumes"
Apr 22 20:29:59.605638 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:29:59.605591    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq" podUID="dc3a80d3-4c53-4eaf-b460-9bc232d9f5ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Apr 22 20:30:00.322114 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:00.322069    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv" podUID="abeff13e-d81d-45be-9051-225f25903669" containerName="sequence-graph-b78d5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:30:04.709722 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:04.709675    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9" podUID="cffff8c4-5712-4f2f-ae2a-a930738658a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Apr 22 20:30:05.322038 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:05.321943    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv" podUID="abeff13e-d81d-45be-9051-225f25903669" containerName="sequence-graph-b78d5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:30:05.322212 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:05.322112    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv"
Apr 22 20:30:09.607098 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:09.607059    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq"
Apr 22 20:30:10.322022 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:10.321979    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv" podUID="abeff13e-d81d-45be-9051-225f25903669" containerName="sequence-graph-b78d5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:30:14.710483 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:14.710408    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9" podUID="cffff8c4-5712-4f2f-ae2a-a930738658a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Apr 22 20:30:15.321723 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:15.321683    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv" podUID="abeff13e-d81d-45be-9051-225f25903669" containerName="sequence-graph-b78d5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:30:20.321614 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:20.321518    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv" podUID="abeff13e-d81d-45be-9051-225f25903669" containerName="sequence-graph-b78d5" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:30:22.802520 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:22.802483    2574 generic.go:358] "Generic (PLEG): container finished" podID="abeff13e-d81d-45be-9051-225f25903669" containerID="e25cb8276cfa9b0df4b1f8165c3e775a5e882f88991dc6f0a933d5361b2ff2c3" exitCode=0
Apr 22 20:30:22.802899 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:22.802553    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv" event={"ID":"abeff13e-d81d-45be-9051-225f25903669","Type":"ContainerDied","Data":"e25cb8276cfa9b0df4b1f8165c3e775a5e882f88991dc6f0a933d5361b2ff2c3"}
Apr 22 20:30:23.419444 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:23.419401    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv"
Apr 22 20:30:23.495948 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:23.495909    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abeff13e-d81d-45be-9051-225f25903669-proxy-tls\") pod \"abeff13e-d81d-45be-9051-225f25903669\" (UID: \"abeff13e-d81d-45be-9051-225f25903669\") "
Apr 22 20:30:23.496135 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:23.495982    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abeff13e-d81d-45be-9051-225f25903669-openshift-service-ca-bundle\") pod \"abeff13e-d81d-45be-9051-225f25903669\" (UID: \"abeff13e-d81d-45be-9051-225f25903669\") "
Apr 22 20:30:23.496359 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:23.496327    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abeff13e-d81d-45be-9051-225f25903669-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "abeff13e-d81d-45be-9051-225f25903669" (UID: "abeff13e-d81d-45be-9051-225f25903669"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:30:23.498137 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:23.498114    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abeff13e-d81d-45be-9051-225f25903669-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "abeff13e-d81d-45be-9051-225f25903669" (UID: "abeff13e-d81d-45be-9051-225f25903669"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:30:23.597514 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:23.597404    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abeff13e-d81d-45be-9051-225f25903669-proxy-tls\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:30:23.597514 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:23.597461    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abeff13e-d81d-45be-9051-225f25903669-openshift-service-ca-bundle\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:30:23.810167 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:23.810130    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv" event={"ID":"abeff13e-d81d-45be-9051-225f25903669","Type":"ContainerDied","Data":"98c93c749f976c267966b582dac1968bad7d3cef3e95492b7575d19d7a6e8270"}
Apr 22 20:30:23.810167 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:23.810164    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv"
Apr 22 20:30:23.810167 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:23.810175    2574 scope.go:117] "RemoveContainer" containerID="e25cb8276cfa9b0df4b1f8165c3e775a5e882f88991dc6f0a933d5361b2ff2c3"
Apr 22 20:30:23.831481 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:23.831445    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv"]
Apr 22 20:30:23.834487 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:23.834453    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-b78d5-6dcd78c574-pfvzv"]
Apr 22 20:30:24.709678 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:24.709632    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9" podUID="cffff8c4-5712-4f2f-ae2a-a930738658a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Apr 22 20:30:25.159543 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:25.159460    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abeff13e-d81d-45be-9051-225f25903669" path="/var/lib/kubelet/pods/abeff13e-d81d-45be-9051-225f25903669/volumes"
Apr 22 20:30:27.353541 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:27.353459    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg"]
Apr 22 20:30:27.353875 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:27.353791    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b0b734e-0537-43a7-a2f6-3f8acf109b02" containerName="kserve-container"
Apr 22 20:30:27.353875 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:27.353817    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b0b734e-0537-43a7-a2f6-3f8acf109b02" containerName="kserve-container"
Apr 22 20:30:27.353875 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:27.353828    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abeff13e-d81d-45be-9051-225f25903669" containerName="sequence-graph-b78d5"
Apr 22 20:30:27.353875 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:27.353833    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="abeff13e-d81d-45be-9051-225f25903669" containerName="sequence-graph-b78d5"
Apr 22 20:30:27.354001 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:27.353885    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="9b0b734e-0537-43a7-a2f6-3f8acf109b02" containerName="kserve-container"
Apr 22 20:30:27.354001 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:27.353897    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="abeff13e-d81d-45be-9051-225f25903669" containerName="sequence-graph-b78d5"
Apr 22 20:30:27.358261 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:27.358238    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg"
Apr 22 20:30:27.361093 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:27.361071    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 22 20:30:27.361242 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:27.361089    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-9d58e-kube-rbac-proxy-sar-config\""
Apr 22 20:30:27.361242 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:27.361078    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-9d58e-serving-cert\""
Apr 22 20:30:27.363955 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:27.363929    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg"]
Apr 22 20:30:27.428245 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:27.428201    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a75108-3c01-4347-8b95-90e30cbc57ef-openshift-service-ca-bundle\") pod \"splitter-graph-9d58e-7f95778cfd-nctcg\" (UID: \"b1a75108-3c01-4347-8b95-90e30cbc57ef\") " pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg"
Apr 22 20:30:27.428452 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:27.428308    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1a75108-3c01-4347-8b95-90e30cbc57ef-proxy-tls\") pod \"splitter-graph-9d58e-7f95778cfd-nctcg\" (UID: \"b1a75108-3c01-4347-8b95-90e30cbc57ef\") " pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg"
Apr 22 20:30:27.528796 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:27.528760    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a75108-3c01-4347-8b95-90e30cbc57ef-openshift-service-ca-bundle\") pod \"splitter-graph-9d58e-7f95778cfd-nctcg\" (UID: \"b1a75108-3c01-4347-8b95-90e30cbc57ef\") " pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg"
Apr 22 20:30:27.528937 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:27.528828    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1a75108-3c01-4347-8b95-90e30cbc57ef-proxy-tls\") pod \"splitter-graph-9d58e-7f95778cfd-nctcg\" (UID: \"b1a75108-3c01-4347-8b95-90e30cbc57ef\") " pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg"
Apr 22 20:30:27.528937 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:30:27.528919    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-9d58e-serving-cert: secret "splitter-graph-9d58e-serving-cert" not found
Apr 22 20:30:27.529007 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:30:27.528982    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a75108-3c01-4347-8b95-90e30cbc57ef-proxy-tls podName:b1a75108-3c01-4347-8b95-90e30cbc57ef nodeName:}" failed. No retries permitted until 2026-04-22 20:30:28.028966789 +0000 UTC m=+1979.446630413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b1a75108-3c01-4347-8b95-90e30cbc57ef-proxy-tls") pod "splitter-graph-9d58e-7f95778cfd-nctcg" (UID: "b1a75108-3c01-4347-8b95-90e30cbc57ef") : secret "splitter-graph-9d58e-serving-cert" not found
Apr 22 20:30:27.529470 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:27.529449    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a75108-3c01-4347-8b95-90e30cbc57ef-openshift-service-ca-bundle\") pod \"splitter-graph-9d58e-7f95778cfd-nctcg\" (UID: \"b1a75108-3c01-4347-8b95-90e30cbc57ef\") " pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg"
Apr 22 20:30:28.032948 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:28.032906    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1a75108-3c01-4347-8b95-90e30cbc57ef-proxy-tls\") pod \"splitter-graph-9d58e-7f95778cfd-nctcg\" (UID: \"b1a75108-3c01-4347-8b95-90e30cbc57ef\") " pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg"
Apr 22 20:30:28.035678 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:28.035656    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1a75108-3c01-4347-8b95-90e30cbc57ef-proxy-tls\") pod \"splitter-graph-9d58e-7f95778cfd-nctcg\" (UID: \"b1a75108-3c01-4347-8b95-90e30cbc57ef\") " pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg"
Apr 22 20:30:28.269880 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:28.269841    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg"
Apr 22 20:30:28.397295 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:28.397256    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg"]
Apr 22 20:30:28.400460 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:30:28.400434    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a75108_3c01_4347_8b95_90e30cbc57ef.slice/crio-7dbfbc4b466b29cc51490e12cf956be5f13b0c0ac05669a9ec89e7d621545b9e WatchSource:0}: Error finding container 7dbfbc4b466b29cc51490e12cf956be5f13b0c0ac05669a9ec89e7d621545b9e: Status 404 returned error can't find the container with id 7dbfbc4b466b29cc51490e12cf956be5f13b0c0ac05669a9ec89e7d621545b9e
Apr 22 20:30:28.831935 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:28.831897    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg" event={"ID":"b1a75108-3c01-4347-8b95-90e30cbc57ef","Type":"ContainerStarted","Data":"26e041e27329b30d068fc9577bcd51e07dbd28b0a46180564070445c499126df"}
Apr 22 20:30:28.831935 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:28.831937    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg" event={"ID":"b1a75108-3c01-4347-8b95-90e30cbc57ef","Type":"ContainerStarted","Data":"7dbfbc4b466b29cc51490e12cf956be5f13b0c0ac05669a9ec89e7d621545b9e"}
Apr 22 20:30:28.832150 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:28.831961    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg"
Apr 22 20:30:28.847628 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:28.847573    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg" podStartSLOduration=1.847554221 podStartE2EDuration="1.847554221s" podCreationTimestamp="2026-04-22 20:30:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:30:28.846588693 +0000 UTC m=+1980.264252365" watchObservedRunningTime="2026-04-22 20:30:28.847554221 +0000 UTC m=+1980.265217866"
Apr 22 20:30:34.709620 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:34.709578    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9" podUID="cffff8c4-5712-4f2f-ae2a-a930738658a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Apr 22 20:30:34.840527 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:34.840493    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg"
Apr 22 20:30:37.429531 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:37.429492    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg"]
Apr 22 20:30:37.429883 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:37.429744    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg" podUID="b1a75108-3c01-4347-8b95-90e30cbc57ef" containerName="splitter-graph-9d58e" containerID="cri-o://26e041e27329b30d068fc9577bcd51e07dbd28b0a46180564070445c499126df" gracePeriod=30
Apr 22 20:30:37.545856 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:37.545817    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc"]
Apr 22 20:30:37.550370 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:37.550344    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc"
Apr 22 20:30:37.554941 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:37.554917    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq"]
Apr 22 20:30:37.555154 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:37.555134    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq" podUID="dc3a80d3-4c53-4eaf-b460-9bc232d9f5ef" containerName="kserve-container" containerID="cri-o://9ba2cad11a6e80905a193ad2620a8784828a94322fab56687a306dd4adebea5a" gracePeriod=30
Apr 22 20:30:37.558955 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:37.558917    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc"]
Apr 22 20:30:37.561445 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:37.561402    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc"
Apr 22 20:30:37.698562 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:37.698537    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc"]
Apr 22 20:30:37.701645 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:30:37.701606    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a58697_8b43_48a0_9f2c_5595a15a3b03.slice/crio-a5dde4acfb1b97a54fddaf31aec6745dedfda4bfce1fc65cbb8049f619639b50 WatchSource:0}: Error finding container a5dde4acfb1b97a54fddaf31aec6745dedfda4bfce1fc65cbb8049f619639b50: Status 404 returned error can't find the container with id a5dde4acfb1b97a54fddaf31aec6745dedfda4bfce1fc65cbb8049f619639b50
Apr 22 20:30:37.858438 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:37.858380    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc" event={"ID":"61a58697-8b43-48a0-9f2c-5595a15a3b03","Type":"ContainerStarted","Data":"4ded48cadbf514ab7ec7ee0fe0cd2b03c1b80ab08e64a8cf53e7bcc181479f15"}
Apr 22 20:30:37.858608 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:37.858446    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc" event={"ID":"61a58697-8b43-48a0-9f2c-5595a15a3b03","Type":"ContainerStarted","Data":"a5dde4acfb1b97a54fddaf31aec6745dedfda4bfce1fc65cbb8049f619639b50"}
Apr 22 20:30:37.874126 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:37.874083    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc" podStartSLOduration=0.87406682 podStartE2EDuration="874.06682ms" podCreationTimestamp="2026-04-22 20:30:37 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:30:37.873612761 +0000 UTC m=+1989.291276408" watchObservedRunningTime="2026-04-22 20:30:37.87406682 +0000 UTC m=+1989.291730467"
Apr 22 20:30:38.861193 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:38.861157    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc"
Apr 22 20:30:38.862645 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:38.862609    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc" podUID="61a58697-8b43-48a0-9f2c-5595a15a3b03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Apr 22 20:30:39.605555 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:39.605508    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq" podUID="dc3a80d3-4c53-4eaf-b460-9bc232d9f5ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Apr 22 20:30:39.839846 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:39.839809    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg" podUID="b1a75108-3c01-4347-8b95-90e30cbc57ef" containerName="splitter-graph-9d58e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:30:39.864248 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:39.864155    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc" podUID="61a58697-8b43-48a0-9f2c-5595a15a3b03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Apr 22 20:30:40.794860 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:40.794835    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq"
Apr 22 20:30:40.867625 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:40.867596    2574 generic.go:358] "Generic (PLEG): container finished" podID="dc3a80d3-4c53-4eaf-b460-9bc232d9f5ef" containerID="9ba2cad11a6e80905a193ad2620a8784828a94322fab56687a306dd4adebea5a" exitCode=0
Apr 22 20:30:40.867977 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:40.867652    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq" event={"ID":"dc3a80d3-4c53-4eaf-b460-9bc232d9f5ef","Type":"ContainerDied","Data":"9ba2cad11a6e80905a193ad2620a8784828a94322fab56687a306dd4adebea5a"}
Apr 22 20:30:40.867977 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:40.867660    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq"
Apr 22 20:30:40.867977 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:40.867679    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq" event={"ID":"dc3a80d3-4c53-4eaf-b460-9bc232d9f5ef","Type":"ContainerDied","Data":"2f1001f62b77a4144c518730f1ac94f510e033232a4d7c885e5b515be1457b8d"}
Apr 22 20:30:40.867977 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:40.867694    2574 scope.go:117] "RemoveContainer" containerID="9ba2cad11a6e80905a193ad2620a8784828a94322fab56687a306dd4adebea5a"
Apr 22 20:30:40.875956 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:40.875934    2574 scope.go:117] "RemoveContainer" containerID="9ba2cad11a6e80905a193ad2620a8784828a94322fab56687a306dd4adebea5a"
Apr 22 20:30:40.876192 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:30:40.876172    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba2cad11a6e80905a193ad2620a8784828a94322fab56687a306dd4adebea5a\": container with ID starting with 9ba2cad11a6e80905a193ad2620a8784828a94322fab56687a306dd4adebea5a not found: ID does not exist" containerID="9ba2cad11a6e80905a193ad2620a8784828a94322fab56687a306dd4adebea5a"
Apr 22 20:30:40.876248 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:40.876202    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba2cad11a6e80905a193ad2620a8784828a94322fab56687a306dd4adebea5a"} err="failed to get container status \"9ba2cad11a6e80905a193ad2620a8784828a94322fab56687a306dd4adebea5a\": rpc error: code = NotFound desc = could not find container \"9ba2cad11a6e80905a193ad2620a8784828a94322fab56687a306dd4adebea5a\": container with ID starting with 9ba2cad11a6e80905a193ad2620a8784828a94322fab56687a306dd4adebea5a not found: ID does not exist"
Apr 22 20:30:40.887017 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:40.886989    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq"]
Apr 22 20:30:40.888900 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:40.888882    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d58e-predictor-6575c449bd-2npcq"]
Apr 22 20:30:41.159394 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:41.159364    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc3a80d3-4c53-4eaf-b460-9bc232d9f5ef" path="/var/lib/kubelet/pods/dc3a80d3-4c53-4eaf-b460-9bc232d9f5ef/volumes"
Apr 22 20:30:44.710604 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:44.710572    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9"
Apr 22 20:30:44.839860 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:44.839824    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg" podUID="b1a75108-3c01-4347-8b95-90e30cbc57ef" containerName="splitter-graph-9d58e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:30:49.839406 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:49.839370    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg" podUID="b1a75108-3c01-4347-8b95-90e30cbc57ef" containerName="splitter-graph-9d58e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:30:49.839847 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:49.839529    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg"
Apr 22 20:30:49.864808 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:49.864779    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc" podUID="61a58697-8b43-48a0-9f2c-5595a15a3b03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Apr 22 20:30:54.839952 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:54.839915    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg" podUID="b1a75108-3c01-4347-8b95-90e30cbc57ef" containerName="splitter-graph-9d58e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:30:59.839058 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:59.839024    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg" podUID="b1a75108-3c01-4347-8b95-90e30cbc57ef" containerName="splitter-graph-9d58e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:30:59.865128 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:30:59.865094    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc" podUID="61a58697-8b43-48a0-9f2c-5595a15a3b03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Apr 22 20:31:02.984526 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:02.984492    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95"]
Apr 22 20:31:02.984990 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:02.984975    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc3a80d3-4c53-4eaf-b460-9bc232d9f5ef" containerName="kserve-container"
Apr 22 20:31:02.985041 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:02.984993    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc3a80d3-4c53-4eaf-b460-9bc232d9f5ef" containerName="kserve-container"
Apr 22 20:31:02.985083 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:02.985076    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="dc3a80d3-4c53-4eaf-b460-9bc232d9f5ef" containerName="kserve-container"
Apr 22 20:31:02.988163 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:02.988146    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95"
Apr 22 20:31:02.990859 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:02.990822    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-b2a29-serving-cert\""
Apr 22 20:31:02.990859 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:02.990853    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-b2a29-kube-rbac-proxy-sar-config\""
Apr 22 20:31:02.993807 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:02.993786    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95"]
Apr 22 20:31:03.004272 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:03.004250    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84067c4a-3051-4f22-a010-8dee5d8c1b92-proxy-tls\") pod \"switch-graph-b2a29-7976f8764f-rxf95\" (UID: \"84067c4a-3051-4f22-a010-8dee5d8c1b92\") " pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95"
Apr 22 20:31:03.004368 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:03.004299    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84067c4a-3051-4f22-a010-8dee5d8c1b92-openshift-service-ca-bundle\") pod \"switch-graph-b2a29-7976f8764f-rxf95\" (UID: \"84067c4a-3051-4f22-a010-8dee5d8c1b92\") " pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95"
Apr 22 20:31:03.105381 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:03.105355    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84067c4a-3051-4f22-a010-8dee5d8c1b92-proxy-tls\") pod \"switch-graph-b2a29-7976f8764f-rxf95\" (UID: \"84067c4a-3051-4f22-a010-8dee5d8c1b92\") " pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95"
Apr 22 20:31:03.105538 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:03.105410    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84067c4a-3051-4f22-a010-8dee5d8c1b92-openshift-service-ca-bundle\") pod \"switch-graph-b2a29-7976f8764f-rxf95\" (UID: \"84067c4a-3051-4f22-a010-8dee5d8c1b92\") " pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95"
Apr 22 20:31:03.105538 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:31:03.105519    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-b2a29-serving-cert: secret "switch-graph-b2a29-serving-cert" not found
Apr 22 20:31:03.105610 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:31:03.105588    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84067c4a-3051-4f22-a010-8dee5d8c1b92-proxy-tls podName:84067c4a-3051-4f22-a010-8dee5d8c1b92 nodeName:}" failed. No retries permitted until 2026-04-22 20:31:03.605572247 +0000 UTC m=+2015.023235871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/84067c4a-3051-4f22-a010-8dee5d8c1b92-proxy-tls") pod "switch-graph-b2a29-7976f8764f-rxf95" (UID: "84067c4a-3051-4f22-a010-8dee5d8c1b92") : secret "switch-graph-b2a29-serving-cert" not found
Apr 22 20:31:03.106046 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:03.106029    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84067c4a-3051-4f22-a010-8dee5d8c1b92-openshift-service-ca-bundle\") pod \"switch-graph-b2a29-7976f8764f-rxf95\" (UID: \"84067c4a-3051-4f22-a010-8dee5d8c1b92\") " pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95"
Apr 22 20:31:03.609538 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:03.609507    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84067c4a-3051-4f22-a010-8dee5d8c1b92-proxy-tls\") pod \"switch-graph-b2a29-7976f8764f-rxf95\" (UID: \"84067c4a-3051-4f22-a010-8dee5d8c1b92\") " pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95"
Apr 22 20:31:03.611909 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:03.611890    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84067c4a-3051-4f22-a010-8dee5d8c1b92-proxy-tls\") pod \"switch-graph-b2a29-7976f8764f-rxf95\" (UID: \"84067c4a-3051-4f22-a010-8dee5d8c1b92\") " pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95"
Apr 22 20:31:03.900489 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:03.900387    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95"
Apr 22 20:31:04.019632 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:04.019604    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95"]
Apr 22 20:31:04.021382 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:31:04.021356    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84067c4a_3051_4f22_a010_8dee5d8c1b92.slice/crio-9ee4314327e0517f816c4b04125d7843efecf094e5f3d520f3787b55c7e7269e WatchSource:0}: Error finding container 9ee4314327e0517f816c4b04125d7843efecf094e5f3d520f3787b55c7e7269e: Status 404 returned error can't find the container with id 9ee4314327e0517f816c4b04125d7843efecf094e5f3d520f3787b55c7e7269e
Apr 22 20:31:04.840091 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:04.840055    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg" podUID="b1a75108-3c01-4347-8b95-90e30cbc57ef" containerName="splitter-graph-9d58e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:31:04.936192 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:04.936154    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95" event={"ID":"84067c4a-3051-4f22-a010-8dee5d8c1b92","Type":"ContainerStarted","Data":"8a533eab169090d879262a7a8251f64dd2809ba04418c71f903060ee2368f13f"}
Apr 22 20:31:04.936345 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:04.936196    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95" event={"ID":"84067c4a-3051-4f22-a010-8dee5d8c1b92","Type":"ContainerStarted","Data":"9ee4314327e0517f816c4b04125d7843efecf094e5f3d520f3787b55c7e7269e"}
Apr 22 20:31:04.936345 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:04.936245    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95"
Apr 22 20:31:04.952105 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:04.952062    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95" podStartSLOduration=2.952048543 podStartE2EDuration="2.952048543s" podCreationTimestamp="2026-04-22 20:31:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:31:04.950329823 +0000 UTC m=+2016.367993481" watchObservedRunningTime="2026-04-22 20:31:04.952048543 +0000 UTC m=+2016.369712188"
Apr 22 20:31:07.573366 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:07.573340    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg"
Apr 22 20:31:07.636885 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:07.636858    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1a75108-3c01-4347-8b95-90e30cbc57ef-proxy-tls\") pod \"b1a75108-3c01-4347-8b95-90e30cbc57ef\" (UID: \"b1a75108-3c01-4347-8b95-90e30cbc57ef\") "
Apr 22 20:31:07.637017 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:07.636962    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a75108-3c01-4347-8b95-90e30cbc57ef-openshift-service-ca-bundle\") pod \"b1a75108-3c01-4347-8b95-90e30cbc57ef\" (UID: \"b1a75108-3c01-4347-8b95-90e30cbc57ef\") "
Apr 22 20:31:07.637281 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:07.637259    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a75108-3c01-4347-8b95-90e30cbc57ef-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "b1a75108-3c01-4347-8b95-90e30cbc57ef" (UID: "b1a75108-3c01-4347-8b95-90e30cbc57ef"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:31:07.638982 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:07.638958    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a75108-3c01-4347-8b95-90e30cbc57ef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b1a75108-3c01-4347-8b95-90e30cbc57ef" (UID: "b1a75108-3c01-4347-8b95-90e30cbc57ef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:31:07.737445 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:07.737392    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a75108-3c01-4347-8b95-90e30cbc57ef-openshift-service-ca-bundle\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:31:07.737445 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:07.737448    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1a75108-3c01-4347-8b95-90e30cbc57ef-proxy-tls\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:31:07.946955 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:07.946918    2574 generic.go:358] "Generic (PLEG): container finished" podID="b1a75108-3c01-4347-8b95-90e30cbc57ef" containerID="26e041e27329b30d068fc9577bcd51e07dbd28b0a46180564070445c499126df" exitCode=0
Apr 22 20:31:07.947107 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:07.946964    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg" event={"ID":"b1a75108-3c01-4347-8b95-90e30cbc57ef","Type":"ContainerDied","Data":"26e041e27329b30d068fc9577bcd51e07dbd28b0a46180564070445c499126df"}
Apr 22 20:31:07.947107 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:07.946992    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg" event={"ID":"b1a75108-3c01-4347-8b95-90e30cbc57ef","Type":"ContainerDied","Data":"7dbfbc4b466b29cc51490e12cf956be5f13b0c0ac05669a9ec89e7d621545b9e"}
Apr 22 20:31:07.947107 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:07.946992    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg"
Apr 22 20:31:07.947107 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:07.947006    2574 scope.go:117] "RemoveContainer" containerID="26e041e27329b30d068fc9577bcd51e07dbd28b0a46180564070445c499126df"
Apr 22 20:31:07.955289 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:07.955270    2574 scope.go:117] "RemoveContainer" containerID="26e041e27329b30d068fc9577bcd51e07dbd28b0a46180564070445c499126df"
Apr 22 20:31:07.955576 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:31:07.955553    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e041e27329b30d068fc9577bcd51e07dbd28b0a46180564070445c499126df\": container with ID starting with 26e041e27329b30d068fc9577bcd51e07dbd28b0a46180564070445c499126df not found: ID does not exist" containerID="26e041e27329b30d068fc9577bcd51e07dbd28b0a46180564070445c499126df"
Apr 22 20:31:07.955645 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:07.955589    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e041e27329b30d068fc9577bcd51e07dbd28b0a46180564070445c499126df"} err="failed to get container status \"26e041e27329b30d068fc9577bcd51e07dbd28b0a46180564070445c499126df\": rpc error: code = NotFound desc = could not find container \"26e041e27329b30d068fc9577bcd51e07dbd28b0a46180564070445c499126df\": container with ID starting with 26e041e27329b30d068fc9577bcd51e07dbd28b0a46180564070445c499126df not found: ID does not exist"
Apr 22 20:31:07.967011 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:07.966968    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg"]
Apr 22 20:31:07.969053 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:07.969033    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9d58e-7f95778cfd-nctcg"]
Apr 22 20:31:09.159580 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:09.159553    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1a75108-3c01-4347-8b95-90e30cbc57ef" path="/var/lib/kubelet/pods/b1a75108-3c01-4347-8b95-90e30cbc57ef/volumes"
Apr 22 20:31:09.864255 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:09.864211    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc" podUID="61a58697-8b43-48a0-9f2c-5595a15a3b03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Apr 22 20:31:10.945772 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:10.945734    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95"
Apr 22 20:31:19.865067 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:19.865029    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc" podUID="61a58697-8b43-48a0-9f2c-5595a15a3b03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Apr 22 20:31:29.865607 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:29.865569    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc"
Apr 22 20:31:47.644011 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:47.643932    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622"]
Apr 22 20:31:47.644471 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:47.644281    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a75108-3c01-4347-8b95-90e30cbc57ef" containerName="splitter-graph-9d58e"
Apr 22 20:31:47.644471 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:47.644293    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a75108-3c01-4347-8b95-90e30cbc57ef" containerName="splitter-graph-9d58e"
Apr 22 20:31:47.644471 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:47.644343    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="b1a75108-3c01-4347-8b95-90e30cbc57ef" containerName="splitter-graph-9d58e"
Apr 22 20:31:47.648750 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:47.648731    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622"
Apr 22 20:31:47.652059 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:47.652034    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-61b05-kube-rbac-proxy-sar-config\""
Apr 22 20:31:47.652203 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:47.652059    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-61b05-serving-cert\""
Apr 22 20:31:47.653703 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:47.653681    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622"]
Apr 22 20:31:47.729714 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:47.729678    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4e2e09e-9cf4-4682-a961-6b3757c15a79-openshift-service-ca-bundle\") pod \"splitter-graph-61b05-7d4d4d4dbf-nf622\" (UID: \"c4e2e09e-9cf4-4682-a961-6b3757c15a79\") " pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622"
Apr 22 20:31:47.729902 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:47.729729    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4e2e09e-9cf4-4682-a961-6b3757c15a79-proxy-tls\") pod \"splitter-graph-61b05-7d4d4d4dbf-nf622\" (UID: \"c4e2e09e-9cf4-4682-a961-6b3757c15a79\") " pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622"
Apr 22 20:31:47.830597 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:47.830551    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4e2e09e-9cf4-4682-a961-6b3757c15a79-openshift-service-ca-bundle\") pod \"splitter-graph-61b05-7d4d4d4dbf-nf622\" (UID: \"c4e2e09e-9cf4-4682-a961-6b3757c15a79\") " pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622"
Apr 22 20:31:47.830788 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:47.830608    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4e2e09e-9cf4-4682-a961-6b3757c15a79-proxy-tls\") pod \"splitter-graph-61b05-7d4d4d4dbf-nf622\" (UID: \"c4e2e09e-9cf4-4682-a961-6b3757c15a79\") " pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622"
Apr 22 20:31:47.830788 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:31:47.830769    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-61b05-serving-cert: secret "splitter-graph-61b05-serving-cert" not found
Apr 22 20:31:47.830916 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:31:47.830847    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4e2e09e-9cf4-4682-a961-6b3757c15a79-proxy-tls podName:c4e2e09e-9cf4-4682-a961-6b3757c15a79 nodeName:}" failed. No retries permitted until 2026-04-22 20:31:48.330825493 +0000 UTC m=+2059.748489118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c4e2e09e-9cf4-4682-a961-6b3757c15a79-proxy-tls") pod "splitter-graph-61b05-7d4d4d4dbf-nf622" (UID: "c4e2e09e-9cf4-4682-a961-6b3757c15a79") : secret "splitter-graph-61b05-serving-cert" not found
Apr 22 20:31:47.831246 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:47.831226    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4e2e09e-9cf4-4682-a961-6b3757c15a79-openshift-service-ca-bundle\") pod \"splitter-graph-61b05-7d4d4d4dbf-nf622\" (UID: \"c4e2e09e-9cf4-4682-a961-6b3757c15a79\") " pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622"
Apr 22 20:31:48.334746 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:48.334711    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4e2e09e-9cf4-4682-a961-6b3757c15a79-proxy-tls\") pod \"splitter-graph-61b05-7d4d4d4dbf-nf622\" (UID: \"c4e2e09e-9cf4-4682-a961-6b3757c15a79\") " pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622"
Apr 22 20:31:48.337127 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:48.337108    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4e2e09e-9cf4-4682-a961-6b3757c15a79-proxy-tls\") pod \"splitter-graph-61b05-7d4d4d4dbf-nf622\" (UID: \"c4e2e09e-9cf4-4682-a961-6b3757c15a79\") " pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622"
Apr 22 20:31:48.559678 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:48.559646    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622"
Apr 22 20:31:48.677593 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:48.677525    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622"]
Apr 22 20:31:48.680286 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:31:48.680257    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e2e09e_9cf4_4682_a961_6b3757c15a79.slice/crio-9f3c5f2f52643f1cf427c659bfe88953dfe659496e0519f4890611a8461ee03d WatchSource:0}: Error finding container 9f3c5f2f52643f1cf427c659bfe88953dfe659496e0519f4890611a8461ee03d: Status 404 returned error can't find the container with id 9f3c5f2f52643f1cf427c659bfe88953dfe659496e0519f4890611a8461ee03d
Apr 22 20:31:49.064236 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:49.064198    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622" event={"ID":"c4e2e09e-9cf4-4682-a961-6b3757c15a79","Type":"ContainerStarted","Data":"2da6b7dafd4332ac519e5a7d652d68dbaa5eda666b0634895af3f388250404be"}
Apr 22 20:31:49.064236 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:49.064236    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622" event={"ID":"c4e2e09e-9cf4-4682-a961-6b3757c15a79","Type":"ContainerStarted","Data":"9f3c5f2f52643f1cf427c659bfe88953dfe659496e0519f4890611a8461ee03d"}
Apr 22 20:31:49.064538 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:49.064334    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622"
Apr 22 20:31:49.078955 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:49.078909    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622" podStartSLOduration=2.0788922850000002 podStartE2EDuration="2.078892285s" podCreationTimestamp="2026-04-22 20:31:47 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:31:49.07837043 +0000 UTC m=+2060.496034087" watchObservedRunningTime="2026-04-22 20:31:49.078892285 +0000 UTC m=+2060.496555930"
Apr 22 20:31:55.072708 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:31:55.072680    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622"
Apr 22 20:40:02.379016 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:02.378982    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622"]
Apr 22 20:40:02.381559 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:02.379220    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622" podUID="c4e2e09e-9cf4-4682-a961-6b3757c15a79" containerName="splitter-graph-61b05" containerID="cri-o://2da6b7dafd4332ac519e5a7d652d68dbaa5eda666b0634895af3f388250404be" gracePeriod=30
Apr 22 20:40:02.465277 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:02.465236    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc"]
Apr 22 20:40:02.465510 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:02.465488    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc" podUID="61a58697-8b43-48a0-9f2c-5595a15a3b03" containerName="kserve-container" containerID="cri-o://4ded48cadbf514ab7ec7ee0fe0cd2b03c1b80ab08e64a8cf53e7bcc181479f15" gracePeriod=30
Apr 22 20:40:05.071523 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:05.071483    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622" podUID="c4e2e09e-9cf4-4682-a961-6b3757c15a79" containerName="splitter-graph-61b05" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:40:05.313161 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:05.313137    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc"
Apr 22 20:40:05.458050 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:05.458015    2574 generic.go:358] "Generic (PLEG): container finished" podID="61a58697-8b43-48a0-9f2c-5595a15a3b03" containerID="4ded48cadbf514ab7ec7ee0fe0cd2b03c1b80ab08e64a8cf53e7bcc181479f15" exitCode=0
Apr 22 20:40:05.458196 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:05.458062    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc" event={"ID":"61a58697-8b43-48a0-9f2c-5595a15a3b03","Type":"ContainerDied","Data":"4ded48cadbf514ab7ec7ee0fe0cd2b03c1b80ab08e64a8cf53e7bcc181479f15"}
Apr 22 20:40:05.458196 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:05.458081    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc"
Apr 22 20:40:05.458196 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:05.458090    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc" event={"ID":"61a58697-8b43-48a0-9f2c-5595a15a3b03","Type":"ContainerDied","Data":"a5dde4acfb1b97a54fddaf31aec6745dedfda4bfce1fc65cbb8049f619639b50"}
Apr 22 20:40:05.458196 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:05.458109    2574 scope.go:117] "RemoveContainer" containerID="4ded48cadbf514ab7ec7ee0fe0cd2b03c1b80ab08e64a8cf53e7bcc181479f15"
Apr 22 20:40:05.466292 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:05.466274    2574 scope.go:117] "RemoveContainer" containerID="4ded48cadbf514ab7ec7ee0fe0cd2b03c1b80ab08e64a8cf53e7bcc181479f15"
Apr 22 20:40:05.466572 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:40:05.466552    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ded48cadbf514ab7ec7ee0fe0cd2b03c1b80ab08e64a8cf53e7bcc181479f15\": container with ID starting with 4ded48cadbf514ab7ec7ee0fe0cd2b03c1b80ab08e64a8cf53e7bcc181479f15 not found: ID does not exist" containerID="4ded48cadbf514ab7ec7ee0fe0cd2b03c1b80ab08e64a8cf53e7bcc181479f15"
Apr 22 20:40:05.466639 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:05.466582    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ded48cadbf514ab7ec7ee0fe0cd2b03c1b80ab08e64a8cf53e7bcc181479f15"} err="failed to get container status \"4ded48cadbf514ab7ec7ee0fe0cd2b03c1b80ab08e64a8cf53e7bcc181479f15\": rpc error: code = NotFound desc = could not find container \"4ded48cadbf514ab7ec7ee0fe0cd2b03c1b80ab08e64a8cf53e7bcc181479f15\": container with ID starting with 4ded48cadbf514ab7ec7ee0fe0cd2b03c1b80ab08e64a8cf53e7bcc181479f15 not found: ID does not exist"
Apr 22 20:40:05.478909 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:05.478887    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc"]
Apr 22 20:40:05.482493 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:05.482470    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-61b05-predictor-57898b97c8-262jc"]
Apr 22 20:40:07.159597 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:07.159559    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61a58697-8b43-48a0-9f2c-5595a15a3b03" path="/var/lib/kubelet/pods/61a58697-8b43-48a0-9f2c-5595a15a3b03/volumes"
Apr 22 20:40:10.071282 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:10.071245    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622" podUID="c4e2e09e-9cf4-4682-a961-6b3757c15a79" containerName="splitter-graph-61b05" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:40:15.070847 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:15.070794    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622" podUID="c4e2e09e-9cf4-4682-a961-6b3757c15a79" containerName="splitter-graph-61b05" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:40:15.071204 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:15.070927    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622"
Apr 22 20:40:20.071328 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:20.071281    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622" podUID="c4e2e09e-9cf4-4682-a961-6b3757c15a79" containerName="splitter-graph-61b05" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:40:25.071631 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:25.071581    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622" podUID="c4e2e09e-9cf4-4682-a961-6b3757c15a79" containerName="splitter-graph-61b05" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:40:30.070692 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:30.070649    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622" podUID="c4e2e09e-9cf4-4682-a961-6b3757c15a79" containerName="splitter-graph-61b05" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:40:32.535635 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:32.535602    2574 generic.go:358] "Generic (PLEG): container finished" podID="c4e2e09e-9cf4-4682-a961-6b3757c15a79" containerID="2da6b7dafd4332ac519e5a7d652d68dbaa5eda666b0634895af3f388250404be" exitCode=0
Apr 22 20:40:32.535966 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:32.535674    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622" event={"ID":"c4e2e09e-9cf4-4682-a961-6b3757c15a79","Type":"ContainerDied","Data":"2da6b7dafd4332ac519e5a7d652d68dbaa5eda666b0634895af3f388250404be"}
Apr 22 20:40:32.535966 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:32.535718    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622" event={"ID":"c4e2e09e-9cf4-4682-a961-6b3757c15a79","Type":"ContainerDied","Data":"9f3c5f2f52643f1cf427c659bfe88953dfe659496e0519f4890611a8461ee03d"}
Apr 22 20:40:32.535966 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:32.535734    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f3c5f2f52643f1cf427c659bfe88953dfe659496e0519f4890611a8461ee03d"
Apr 22 20:40:32.543212 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:32.543196    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622"
Apr 22 20:40:32.701244 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:32.701159    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4e2e09e-9cf4-4682-a961-6b3757c15a79-proxy-tls\") pod \"c4e2e09e-9cf4-4682-a961-6b3757c15a79\" (UID: \"c4e2e09e-9cf4-4682-a961-6b3757c15a79\") "
Apr 22 20:40:32.701401 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:32.701295    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4e2e09e-9cf4-4682-a961-6b3757c15a79-openshift-service-ca-bundle\") pod \"c4e2e09e-9cf4-4682-a961-6b3757c15a79\" (UID: \"c4e2e09e-9cf4-4682-a961-6b3757c15a79\") "
Apr 22 20:40:32.701677 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:32.701651    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e2e09e-9cf4-4682-a961-6b3757c15a79-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c4e2e09e-9cf4-4682-a961-6b3757c15a79" (UID: "c4e2e09e-9cf4-4682-a961-6b3757c15a79"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:40:32.703249 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:32.703228    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4e2e09e-9cf4-4682-a961-6b3757c15a79-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c4e2e09e-9cf4-4682-a961-6b3757c15a79" (UID: "c4e2e09e-9cf4-4682-a961-6b3757c15a79"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:40:32.802510 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:32.802486    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4e2e09e-9cf4-4682-a961-6b3757c15a79-proxy-tls\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:40:32.802510 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:32.802507    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4e2e09e-9cf4-4682-a961-6b3757c15a79-openshift-service-ca-bundle\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:40:33.538605 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:33.538578    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622"
Apr 22 20:40:33.553976 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:33.553946    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622"]
Apr 22 20:40:33.557353 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:33.557329    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-61b05-7d4d4d4dbf-nf622"]
Apr 22 20:40:35.159443 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:40:35.159380    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4e2e09e-9cf4-4682-a961-6b3757c15a79" path="/var/lib/kubelet/pods/c4e2e09e-9cf4-4682-a961-6b3757c15a79/volumes"
Apr 22 20:41:29.269297 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:41:29.269260    2574 scope.go:117] "RemoveContainer" containerID="2da6b7dafd4332ac519e5a7d652d68dbaa5eda666b0634895af3f388250404be"
Apr 22 20:47:22.412358 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:22.412324    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95"]
Apr 22 20:47:22.414928 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:22.412576    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95" podUID="84067c4a-3051-4f22-a010-8dee5d8c1b92" containerName="switch-graph-b2a29" containerID="cri-o://8a533eab169090d879262a7a8251f64dd2809ba04418c71f903060ee2368f13f" gracePeriod=30
Apr 22 20:47:22.512070 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:22.512032    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9"]
Apr 22 20:47:22.512386 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:22.512337    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9" podUID="cffff8c4-5712-4f2f-ae2a-a930738658a8" containerName="kserve-container" containerID="cri-o://2ae8cc2fd110eb363ef00eb480b5e9f5800a67419a1699e0c710a9bc97fc4189" gracePeriod=30
Apr 22 20:47:23.387233 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:23.387190    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5825m/must-gather-vlpb6"]
Apr 22 20:47:23.387566 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:23.387539    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a58697-8b43-48a0-9f2c-5595a15a3b03" containerName="kserve-container"
Apr 22 20:47:23.387566 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:23.387557    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a58697-8b43-48a0-9f2c-5595a15a3b03" containerName="kserve-container"
Apr 22 20:47:23.387792 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:23.387580    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4e2e09e-9cf4-4682-a961-6b3757c15a79" containerName="splitter-graph-61b05"
Apr 22 20:47:23.387792 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:23.387586    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4e2e09e-9cf4-4682-a961-6b3757c15a79" containerName="splitter-graph-61b05"
Apr 22 20:47:23.387792 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:23.387650    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="c4e2e09e-9cf4-4682-a961-6b3757c15a79" containerName="splitter-graph-61b05"
Apr 22 20:47:23.387792 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:23.387664    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="61a58697-8b43-48a0-9f2c-5595a15a3b03" containerName="kserve-container"
Apr 22 20:47:23.390836 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:23.390819    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5825m/must-gather-vlpb6"
Apr 22 20:47:23.393522 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:23.393500    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5825m\"/\"openshift-service-ca.crt\""
Apr 22 20:47:23.394741 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:23.394720    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5825m\"/\"kube-root-ca.crt\""
Apr 22 20:47:23.394852 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:23.394775    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5825m\"/\"default-dockercfg-ggsv5\""
Apr 22 20:47:23.403609 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:23.403591    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5825m/must-gather-vlpb6"]
Apr 22 20:47:23.418564 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:23.418544    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkvpg\" (UniqueName: \"kubernetes.io/projected/011522ea-7372-40d1-b099-775a905d3384-kube-api-access-xkvpg\") pod \"must-gather-vlpb6\" (UID: \"011522ea-7372-40d1-b099-775a905d3384\") " pod="openshift-must-gather-5825m/must-gather-vlpb6"
Apr 22 20:47:23.418819 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:23.418574    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/011522ea-7372-40d1-b099-775a905d3384-must-gather-output\") pod \"must-gather-vlpb6\" (UID: \"011522ea-7372-40d1-b099-775a905d3384\") " pod="openshift-must-gather-5825m/must-gather-vlpb6"
Apr 22 20:47:23.519674 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:23.519646    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/011522ea-7372-40d1-b099-775a905d3384-must-gather-output\") pod \"must-gather-vlpb6\" (UID: \"011522ea-7372-40d1-b099-775a905d3384\") " pod="openshift-must-gather-5825m/must-gather-vlpb6"
Apr 22 20:47:23.519790 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:23.519745    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xkvpg\" (UniqueName: \"kubernetes.io/projected/011522ea-7372-40d1-b099-775a905d3384-kube-api-access-xkvpg\") pod \"must-gather-vlpb6\" (UID: \"011522ea-7372-40d1-b099-775a905d3384\") " pod="openshift-must-gather-5825m/must-gather-vlpb6"
Apr 22 20:47:23.520035 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:23.520017    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/011522ea-7372-40d1-b099-775a905d3384-must-gather-output\") pod \"must-gather-vlpb6\" (UID: \"011522ea-7372-40d1-b099-775a905d3384\") " pod="openshift-must-gather-5825m/must-gather-vlpb6"
Apr 22 20:47:23.529730 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:23.529703    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkvpg\" (UniqueName: \"kubernetes.io/projected/011522ea-7372-40d1-b099-775a905d3384-kube-api-access-xkvpg\") pod \"must-gather-vlpb6\" (UID: \"011522ea-7372-40d1-b099-775a905d3384\") " pod="openshift-must-gather-5825m/must-gather-vlpb6"
Apr 22 20:47:23.713531 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:23.713497    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5825m/must-gather-vlpb6"
Apr 22 20:47:23.836153 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:23.836130    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5825m/must-gather-vlpb6"]
Apr 22 20:47:23.838316 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:47:23.838286    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod011522ea_7372_40d1_b099_775a905d3384.slice/crio-3c74d114df77177d9e63becc7f494f60858e3bd60b632cd5f1e62424d3627a7f WatchSource:0}: Error finding container 3c74d114df77177d9e63becc7f494f60858e3bd60b632cd5f1e62424d3627a7f: Status 404 returned error can't find the container with id 3c74d114df77177d9e63becc7f494f60858e3bd60b632cd5f1e62424d3627a7f
Apr 22 20:47:23.840005 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:23.839987    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 22 20:47:24.669162 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:24.669127    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5825m/must-gather-vlpb6" event={"ID":"011522ea-7372-40d1-b099-775a905d3384","Type":"ContainerStarted","Data":"3c74d114df77177d9e63becc7f494f60858e3bd60b632cd5f1e62424d3627a7f"}
Apr 22 20:47:24.709618 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:24.709588    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9" podUID="cffff8c4-5712-4f2f-ae2a-a930738658a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Apr 22 20:47:25.674346 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:25.674310    2574 generic.go:358] "Generic (PLEG): container finished" podID="cffff8c4-5712-4f2f-ae2a-a930738658a8" containerID="2ae8cc2fd110eb363ef00eb480b5e9f5800a67419a1699e0c710a9bc97fc4189" exitCode=0
Apr 22 20:47:25.674778 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:25.674453    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9" event={"ID":"cffff8c4-5712-4f2f-ae2a-a930738658a8","Type":"ContainerDied","Data":"2ae8cc2fd110eb363ef00eb480b5e9f5800a67419a1699e0c710a9bc97fc4189"}
Apr 22 20:47:25.777971 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:25.777949    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9"
Apr 22 20:47:25.943736 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:25.943701    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95" podUID="84067c4a-3051-4f22-a010-8dee5d8c1b92" containerName="switch-graph-b2a29" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:47:26.678706 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:26.678667    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9" event={"ID":"cffff8c4-5712-4f2f-ae2a-a930738658a8","Type":"ContainerDied","Data":"fd3b5117ec4bca9fe95a3d6c0c93d4f9596572c2bb3965e0458227289ffe2584"}
Apr 22 20:47:26.679168 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:26.678726    2574 scope.go:117] "RemoveContainer" containerID="2ae8cc2fd110eb363ef00eb480b5e9f5800a67419a1699e0c710a9bc97fc4189"
Apr 22 20:47:26.679168 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:26.678686    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9"
Apr 22 20:47:26.700022 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:26.699995    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9"]
Apr 22 20:47:26.704250 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:26.704229    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b2a29-predictor-7797c69846-h6mp9"]
Apr 22 20:47:27.160975 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:27.160940    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cffff8c4-5712-4f2f-ae2a-a930738658a8" path="/var/lib/kubelet/pods/cffff8c4-5712-4f2f-ae2a-a930738658a8/volumes"
Apr 22 20:47:28.691738 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:28.691692    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5825m/must-gather-vlpb6" event={"ID":"011522ea-7372-40d1-b099-775a905d3384","Type":"ContainerStarted","Data":"7395e76417f5916f8db1117f14a818e2997d33999ccffd066aa88ef6fc12d4ac"}
Apr 22 20:47:28.691738 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:28.691736    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5825m/must-gather-vlpb6" event={"ID":"011522ea-7372-40d1-b099-775a905d3384","Type":"ContainerStarted","Data":"c20c877d96e4d809b6cc656593beab403be05a0f6b0a77720a584f97832de608"}
Apr 22 20:47:28.707259 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:28.707192    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-5825m/must-gather-vlpb6" podStartSLOduration=1.350661473 podStartE2EDuration="5.707174781s" podCreationTimestamp="2026-04-22 20:47:23 +0000 UTC" firstStartedPulling="2026-04-22 20:47:23.84011685 +0000 UTC m=+2995.257780474" lastFinishedPulling="2026-04-22 20:47:28.196630145 +0000 UTC m=+2999.614293782" observedRunningTime="2026-04-22 20:47:28.706519586 +0000 UTC m=+3000.124183232" watchObservedRunningTime="2026-04-22 20:47:28.707174781 +0000 UTC m=+3000.124838428"
Apr 22 20:47:30.944998 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:30.944958    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95" podUID="84067c4a-3051-4f22-a010-8dee5d8c1b92" containerName="switch-graph-b2a29" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:47:35.944198 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:35.944152    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95" podUID="84067c4a-3051-4f22-a010-8dee5d8c1b92" containerName="switch-graph-b2a29" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:47:35.944611 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:35.944260    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95"
Apr 22 20:47:36.933668 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:36.933634    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2a29-7976f8764f-rxf95_84067c4a-3051-4f22-a010-8dee5d8c1b92/switch-graph-b2a29/0.log"
Apr 22 20:47:37.629720 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:37.629693    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2a29-7976f8764f-rxf95_84067c4a-3051-4f22-a010-8dee5d8c1b92/switch-graph-b2a29/0.log"
Apr 22 20:47:38.338219 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:38.338194    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2a29-7976f8764f-rxf95_84067c4a-3051-4f22-a010-8dee5d8c1b92/switch-graph-b2a29/0.log"
Apr 22 20:47:39.027906 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:39.027875    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2a29-7976f8764f-rxf95_84067c4a-3051-4f22-a010-8dee5d8c1b92/switch-graph-b2a29/0.log"
Apr 22 20:47:39.737144 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:39.737115    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2a29-7976f8764f-rxf95_84067c4a-3051-4f22-a010-8dee5d8c1b92/switch-graph-b2a29/0.log"
Apr 22 20:47:40.418596 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:40.418557    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2a29-7976f8764f-rxf95_84067c4a-3051-4f22-a010-8dee5d8c1b92/switch-graph-b2a29/0.log"
Apr 22 20:47:40.944047 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:40.944002    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95" podUID="84067c4a-3051-4f22-a010-8dee5d8c1b92" containerName="switch-graph-b2a29" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:47:41.113409 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:41.113367    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2a29-7976f8764f-rxf95_84067c4a-3051-4f22-a010-8dee5d8c1b92/switch-graph-b2a29/0.log"
Apr 22 20:47:41.787374 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:41.787342    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2a29-7976f8764f-rxf95_84067c4a-3051-4f22-a010-8dee5d8c1b92/switch-graph-b2a29/0.log"
Apr 22 20:47:42.477873 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:42.477836    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2a29-7976f8764f-rxf95_84067c4a-3051-4f22-a010-8dee5d8c1b92/switch-graph-b2a29/0.log"
Apr 22 20:47:43.173388 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:43.173356    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2a29-7976f8764f-rxf95_84067c4a-3051-4f22-a010-8dee5d8c1b92/switch-graph-b2a29/0.log"
Apr 22 20:47:43.878568 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:43.878541    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2a29-7976f8764f-rxf95_84067c4a-3051-4f22-a010-8dee5d8c1b92/switch-graph-b2a29/0.log"
Apr 22 20:47:44.587654 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:44.587624    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-b2a29-7976f8764f-rxf95_84067c4a-3051-4f22-a010-8dee5d8c1b92/switch-graph-b2a29/0.log"
Apr 22 20:47:45.943978 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:45.943939    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95" podUID="84067c4a-3051-4f22-a010-8dee5d8c1b92" containerName="switch-graph-b2a29" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:47:46.745667 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:46.745635    2574 generic.go:358] "Generic (PLEG): container finished" podID="011522ea-7372-40d1-b099-775a905d3384" containerID="c20c877d96e4d809b6cc656593beab403be05a0f6b0a77720a584f97832de608" exitCode=0
Apr 22 20:47:46.745873 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:46.745705    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5825m/must-gather-vlpb6" event={"ID":"011522ea-7372-40d1-b099-775a905d3384","Type":"ContainerDied","Data":"c20c877d96e4d809b6cc656593beab403be05a0f6b0a77720a584f97832de608"}
Apr 22 20:47:46.746035 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:46.746021    2574 scope.go:117] "RemoveContainer" containerID="c20c877d96e4d809b6cc656593beab403be05a0f6b0a77720a584f97832de608"
Apr 22 20:47:47.081664 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:47.081562    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5825m_must-gather-vlpb6_011522ea-7372-40d1-b099-775a905d3384/gather/0.log"
Apr 22 20:47:50.372701 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:50.372673    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-89zh2_bcfcba31-705d-4865-bba0-528576e44d02/global-pull-secret-syncer/0.log"
Apr 22 20:47:50.580806 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:50.580764    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wmn8m_844c05e7-3e07-4c4a-98d5-359d7da599f2/konnectivity-agent/0.log"
Apr 22 20:47:50.621096 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:50.621069    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-215.ec2.internal_2489d845e4aa755919265a93aff51647/haproxy/0.log"
Apr 22 20:47:50.943391 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:50.943354    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95" podUID="84067c4a-3051-4f22-a010-8dee5d8c1b92" containerName="switch-graph-b2a29" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 22 20:47:52.552937 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.552914    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95"
Apr 22 20:47:52.664908 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.664880    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84067c4a-3051-4f22-a010-8dee5d8c1b92-proxy-tls\") pod \"84067c4a-3051-4f22-a010-8dee5d8c1b92\" (UID: \"84067c4a-3051-4f22-a010-8dee5d8c1b92\") "
Apr 22 20:47:52.665037 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.664935    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84067c4a-3051-4f22-a010-8dee5d8c1b92-openshift-service-ca-bundle\") pod \"84067c4a-3051-4f22-a010-8dee5d8c1b92\" (UID: \"84067c4a-3051-4f22-a010-8dee5d8c1b92\") "
Apr 22 20:47:52.665328 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.665299    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84067c4a-3051-4f22-a010-8dee5d8c1b92-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "84067c4a-3051-4f22-a010-8dee5d8c1b92" (UID: "84067c4a-3051-4f22-a010-8dee5d8c1b92"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 22 20:47:52.667055 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.667011    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84067c4a-3051-4f22-a010-8dee5d8c1b92-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "84067c4a-3051-4f22-a010-8dee5d8c1b92" (UID: "84067c4a-3051-4f22-a010-8dee5d8c1b92"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 22 20:47:52.715956 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.715928    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-5825m/must-gather-vlpb6"]
Apr 22 20:47:52.716140 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.716121    2574 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-5825m/must-gather-vlpb6" podUID="011522ea-7372-40d1-b099-775a905d3384" containerName="copy" containerID="cri-o://7395e76417f5916f8db1117f14a818e2997d33999ccffd066aa88ef6fc12d4ac" gracePeriod=2
Apr 22 20:47:52.722190 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.722167    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-5825m/must-gather-vlpb6"]
Apr 22 20:47:52.763644 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.763614    2574 generic.go:358] "Generic (PLEG): container finished" podID="84067c4a-3051-4f22-a010-8dee5d8c1b92" containerID="8a533eab169090d879262a7a8251f64dd2809ba04418c71f903060ee2368f13f" exitCode=0
Apr 22 20:47:52.763752 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.763683    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95"
Apr 22 20:47:52.763752 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.763698    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95" event={"ID":"84067c4a-3051-4f22-a010-8dee5d8c1b92","Type":"ContainerDied","Data":"8a533eab169090d879262a7a8251f64dd2809ba04418c71f903060ee2368f13f"}
Apr 22 20:47:52.763752 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.763736    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95" event={"ID":"84067c4a-3051-4f22-a010-8dee5d8c1b92","Type":"ContainerDied","Data":"9ee4314327e0517f816c4b04125d7843efecf094e5f3d520f3787b55c7e7269e"}
Apr 22 20:47:52.763861 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.763753    2574 scope.go:117] "RemoveContainer" containerID="8a533eab169090d879262a7a8251f64dd2809ba04418c71f903060ee2368f13f"
Apr 22 20:47:52.765736 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.765718    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84067c4a-3051-4f22-a010-8dee5d8c1b92-proxy-tls\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:47:52.765800 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.765742    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84067c4a-3051-4f22-a010-8dee5d8c1b92-openshift-service-ca-bundle\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:47:52.768752 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.768727    2574 status_manager.go:895] "Failed to get status for pod" podUID="011522ea-7372-40d1-b099-775a905d3384" pod="openshift-must-gather-5825m/must-gather-vlpb6" err="pods \"must-gather-vlpb6\" is forbidden: User \"system:node:ip-10-0-135-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-5825m\": no relationship found between node 'ip-10-0-135-215.ec2.internal' and this object"
Apr 22 20:47:52.804441 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.801515    2574 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95"]
Apr 22 20:47:52.806486 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.806465    2574 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b2a29-7976f8764f-rxf95"]
Apr 22 20:47:52.807034 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.807010    2574 status_manager.go:895] "Failed to get status for pod" podUID="011522ea-7372-40d1-b099-775a905d3384" pod="openshift-must-gather-5825m/must-gather-vlpb6" err="pods \"must-gather-vlpb6\" is forbidden: User \"system:node:ip-10-0-135-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-5825m\": no relationship found between node 'ip-10-0-135-215.ec2.internal' and this object"
Apr 22 20:47:52.826240 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.826219    2574 scope.go:117] "RemoveContainer" containerID="8a533eab169090d879262a7a8251f64dd2809ba04418c71f903060ee2368f13f"
Apr 22 20:47:52.826532 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:47:52.826514    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a533eab169090d879262a7a8251f64dd2809ba04418c71f903060ee2368f13f\": container with ID starting with 8a533eab169090d879262a7a8251f64dd2809ba04418c71f903060ee2368f13f not found: ID does not exist" containerID="8a533eab169090d879262a7a8251f64dd2809ba04418c71f903060ee2368f13f"
Apr 22 20:47:52.826584 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.826539    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a533eab169090d879262a7a8251f64dd2809ba04418c71f903060ee2368f13f"} err="failed to get container status \"8a533eab169090d879262a7a8251f64dd2809ba04418c71f903060ee2368f13f\": rpc error: code = NotFound desc = could not find container \"8a533eab169090d879262a7a8251f64dd2809ba04418c71f903060ee2368f13f\": container with ID starting with 8a533eab169090d879262a7a8251f64dd2809ba04418c71f903060ee2368f13f not found: ID does not exist"
Apr 22 20:47:52.930241 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.930218    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5825m_must-gather-vlpb6_011522ea-7372-40d1-b099-775a905d3384/copy/0.log"
Apr 22 20:47:52.930578 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.930561    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5825m/must-gather-vlpb6"
Apr 22 20:47:52.932617 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:52.932589    2574 status_manager.go:895] "Failed to get status for pod" podUID="011522ea-7372-40d1-b099-775a905d3384" pod="openshift-must-gather-5825m/must-gather-vlpb6" err="pods \"must-gather-vlpb6\" is forbidden: User \"system:node:ip-10-0-135-215.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-5825m\": no relationship found between node 'ip-10-0-135-215.ec2.internal' and this object"
Apr 22 20:47:53.068548 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:53.068523    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/011522ea-7372-40d1-b099-775a905d3384-must-gather-output\") pod \"011522ea-7372-40d1-b099-775a905d3384\" (UID: \"011522ea-7372-40d1-b099-775a905d3384\") "
Apr 22 20:47:53.068688 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:53.068570    2574 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkvpg\" (UniqueName: \"kubernetes.io/projected/011522ea-7372-40d1-b099-775a905d3384-kube-api-access-xkvpg\") pod \"011522ea-7372-40d1-b099-775a905d3384\" (UID: \"011522ea-7372-40d1-b099-775a905d3384\") "
Apr 22 20:47:53.069876 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:53.069851    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011522ea-7372-40d1-b099-775a905d3384-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "011522ea-7372-40d1-b099-775a905d3384" (UID: "011522ea-7372-40d1-b099-775a905d3384"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 22 20:47:53.070888 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:53.070860    2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/011522ea-7372-40d1-b099-775a905d3384-kube-api-access-xkvpg" (OuterVolumeSpecName: "kube-api-access-xkvpg") pod "011522ea-7372-40d1-b099-775a905d3384" (UID: "011522ea-7372-40d1-b099-775a905d3384"). InnerVolumeSpecName "kube-api-access-xkvpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 22 20:47:53.159958 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:53.159933    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="011522ea-7372-40d1-b099-775a905d3384" path="/var/lib/kubelet/pods/011522ea-7372-40d1-b099-775a905d3384/volumes"
Apr 22 20:47:53.160299 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:53.160287    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84067c4a-3051-4f22-a010-8dee5d8c1b92" path="/var/lib/kubelet/pods/84067c4a-3051-4f22-a010-8dee5d8c1b92/volumes"
Apr 22 20:47:53.169018 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:53.168999    2574 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/011522ea-7372-40d1-b099-775a905d3384-must-gather-output\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:47:53.169094 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:53.169021    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkvpg\" (UniqueName: \"kubernetes.io/projected/011522ea-7372-40d1-b099-775a905d3384-kube-api-access-xkvpg\") on node \"ip-10-0-135-215.ec2.internal\" DevicePath \"\""
Apr 22 20:47:53.768722 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:53.768695    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5825m_must-gather-vlpb6_011522ea-7372-40d1-b099-775a905d3384/copy/0.log"
Apr 22 20:47:53.769082 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:53.768995    2574 generic.go:358] "Generic (PLEG): container finished" podID="011522ea-7372-40d1-b099-775a905d3384" containerID="7395e76417f5916f8db1117f14a818e2997d33999ccffd066aa88ef6fc12d4ac" exitCode=143
Apr 22 20:47:53.769082 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:53.769047    2574 scope.go:117] "RemoveContainer" containerID="7395e76417f5916f8db1117f14a818e2997d33999ccffd066aa88ef6fc12d4ac"
Apr 22 20:47:53.769082 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:53.769050    2574 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5825m/must-gather-vlpb6"
Apr 22 20:47:53.775935 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:53.775920    2574 scope.go:117] "RemoveContainer" containerID="c20c877d96e4d809b6cc656593beab403be05a0f6b0a77720a584f97832de608"
Apr 22 20:47:53.787221 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:53.787204    2574 scope.go:117] "RemoveContainer" containerID="7395e76417f5916f8db1117f14a818e2997d33999ccffd066aa88ef6fc12d4ac"
Apr 22 20:47:53.787476 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:47:53.787455    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7395e76417f5916f8db1117f14a818e2997d33999ccffd066aa88ef6fc12d4ac\": container with ID starting with 7395e76417f5916f8db1117f14a818e2997d33999ccffd066aa88ef6fc12d4ac not found: ID does not exist" containerID="7395e76417f5916f8db1117f14a818e2997d33999ccffd066aa88ef6fc12d4ac"
Apr 22 20:47:53.787582 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:53.787481    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7395e76417f5916f8db1117f14a818e2997d33999ccffd066aa88ef6fc12d4ac"} err="failed to get container status \"7395e76417f5916f8db1117f14a818e2997d33999ccffd066aa88ef6fc12d4ac\": rpc error: code = NotFound desc = could not find container \"7395e76417f5916f8db1117f14a818e2997d33999ccffd066aa88ef6fc12d4ac\": container with ID starting with 7395e76417f5916f8db1117f14a818e2997d33999ccffd066aa88ef6fc12d4ac not found: ID does not exist"
Apr 22 20:47:53.787582 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:53.787499    2574 scope.go:117] "RemoveContainer" containerID="c20c877d96e4d809b6cc656593beab403be05a0f6b0a77720a584f97832de608"
Apr 22 20:47:53.787722 ip-10-0-135-215 kubenswrapper[2574]: E0422 20:47:53.787704    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c20c877d96e4d809b6cc656593beab403be05a0f6b0a77720a584f97832de608\": container with ID starting with c20c877d96e4d809b6cc656593beab403be05a0f6b0a77720a584f97832de608 not found: ID does not exist" containerID="c20c877d96e4d809b6cc656593beab403be05a0f6b0a77720a584f97832de608"
Apr 22 20:47:53.787769 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:53.787726    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c20c877d96e4d809b6cc656593beab403be05a0f6b0a77720a584f97832de608"} err="failed to get container status \"c20c877d96e4d809b6cc656593beab403be05a0f6b0a77720a584f97832de608\": rpc error: code = NotFound desc = could not find container \"c20c877d96e4d809b6cc656593beab403be05a0f6b0a77720a584f97832de608\": container with ID starting with c20c877d96e4d809b6cc656593beab403be05a0f6b0a77720a584f97832de608 not found: ID does not exist"
Apr 22 20:47:54.292257 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:54.292223    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-69db897b98-s4j8t_0aefc85e-890b-4111-b151-cdea5415289d/kube-state-metrics/0.log"
Apr 22 20:47:54.310864 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:54.310835    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-69db897b98-s4j8t_0aefc85e-890b-4111-b151-cdea5415289d/kube-rbac-proxy-main/0.log"
Apr 22 20:47:54.337868 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:54.337846    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-69db897b98-s4j8t_0aefc85e-890b-4111-b151-cdea5415289d/kube-rbac-proxy-self/0.log"
Apr 22 20:47:54.578466 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:54.578360    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mbktf_40ed9426-93f6-4d39-af17-0aff9cbfc8b7/node-exporter/0.log"
Apr 22 20:47:54.597542 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:54.597512    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mbktf_40ed9426-93f6-4d39-af17-0aff9cbfc8b7/kube-rbac-proxy/0.log"
Apr 22 20:47:54.617546 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:54.617520    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mbktf_40ed9426-93f6-4d39-af17-0aff9cbfc8b7/init-textfile/0.log"
Apr 22 20:47:54.732096 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:54.732062    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c65017f-d1d5-443b-bf4d-6129241ccc09/prometheus/0.log"
Apr 22 20:47:54.748464 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:54.748438    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c65017f-d1d5-443b-bf4d-6129241ccc09/config-reloader/0.log"
Apr 22 20:47:54.771315 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:54.771295    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c65017f-d1d5-443b-bf4d-6129241ccc09/thanos-sidecar/0.log"
Apr 22 20:47:54.790444 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:54.790405    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c65017f-d1d5-443b-bf4d-6129241ccc09/kube-rbac-proxy-web/0.log"
Apr 22 20:47:54.815071 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:54.815053    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c65017f-d1d5-443b-bf4d-6129241ccc09/kube-rbac-proxy/0.log"
Apr 22 20:47:54.838942 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:54.838879    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c65017f-d1d5-443b-bf4d-6129241ccc09/kube-rbac-proxy-thanos/0.log"
Apr 22 20:47:54.862522 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:54.862503    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c65017f-d1d5-443b-bf4d-6129241ccc09/init-config-reloader/0.log"
Apr 22 20:47:54.931724 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:54.931706    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-57cf98b594-rcmm5_88ede3f5-5ebd-4226-94d9-6a2b8687ebe6/prometheus-operator-admission-webhook/0.log"
Apr 22 20:47:56.222885 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:56.222855    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-cb95c66f6-27n8r_173cdd12-cd9e-403c-a2b2-e18a20be58a4/networking-console-plugin/0.log"
Apr 22 20:47:57.505939 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.505900    2574 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"]
Apr 22 20:47:57.506529 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.506508    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="011522ea-7372-40d1-b099-775a905d3384" containerName="gather"
Apr 22 20:47:57.506628 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.506531    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="011522ea-7372-40d1-b099-775a905d3384" containerName="gather"
Apr 22 20:47:57.506628 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.506563    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cffff8c4-5712-4f2f-ae2a-a930738658a8" containerName="kserve-container"
Apr 22 20:47:57.506628 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.506572    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cffff8c4-5712-4f2f-ae2a-a930738658a8" containerName="kserve-container"
Apr 22 20:47:57.506628 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.506596    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="011522ea-7372-40d1-b099-775a905d3384" containerName="copy"
Apr 22 20:47:57.506628 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.506604    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="011522ea-7372-40d1-b099-775a905d3384" containerName="copy"
Apr 22 20:47:57.506628 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.506622    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84067c4a-3051-4f22-a010-8dee5d8c1b92" containerName="switch-graph-b2a29"
Apr 22 20:47:57.506628 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.506630    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="84067c4a-3051-4f22-a010-8dee5d8c1b92" containerName="switch-graph-b2a29"
Apr 22 20:47:57.506944 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.506730    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="011522ea-7372-40d1-b099-775a905d3384" containerName="copy"
Apr 22 20:47:57.506944 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.506749    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="cffff8c4-5712-4f2f-ae2a-a930738658a8" containerName="kserve-container"
Apr 22 20:47:57.506944 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.506761    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="84067c4a-3051-4f22-a010-8dee5d8c1b92" containerName="switch-graph-b2a29"
Apr 22 20:47:57.506944 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.506775    2574 memory_manager.go:356] "RemoveStaleState removing state" podUID="011522ea-7372-40d1-b099-775a905d3384" containerName="gather"
Apr 22 20:47:57.510504 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.510480    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"
Apr 22 20:47:57.512837 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.512820    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xlhgw\"/\"kube-root-ca.crt\""
Apr 22 20:47:57.513366 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.513350    2574 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xlhgw\"/\"openshift-service-ca.crt\""
Apr 22 20:47:57.514953 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.514934    2574 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xlhgw\"/\"default-dockercfg-pm84g\""
Apr 22 20:47:57.516113 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.516092    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"]
Apr 22 20:47:57.601640 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.601618    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f31679e-6608-47b5-94c9-5c26f6f208e3-sys\") pod \"perf-node-gather-daemonset-wfpkc\" (UID: \"6f31679e-6608-47b5-94c9-5c26f6f208e3\") " pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"
Apr 22 20:47:57.601765 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.601656    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6f31679e-6608-47b5-94c9-5c26f6f208e3-proc\") pod \"perf-node-gather-daemonset-wfpkc\" (UID: \"6f31679e-6608-47b5-94c9-5c26f6f208e3\") " pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"
Apr 22 20:47:57.601765 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.601673    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f31679e-6608-47b5-94c9-5c26f6f208e3-lib-modules\") pod \"perf-node-gather-daemonset-wfpkc\" (UID: \"6f31679e-6608-47b5-94c9-5c26f6f208e3\") " pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"
Apr 22 20:47:57.601882 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.601774    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c28x2\" (UniqueName: \"kubernetes.io/projected/6f31679e-6608-47b5-94c9-5c26f6f208e3-kube-api-access-c28x2\") pod \"perf-node-gather-daemonset-wfpkc\" (UID: \"6f31679e-6608-47b5-94c9-5c26f6f208e3\") " pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"
Apr 22 20:47:57.601882 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.601838    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6f31679e-6608-47b5-94c9-5c26f6f208e3-podres\") pod \"perf-node-gather-daemonset-wfpkc\" (UID: \"6f31679e-6608-47b5-94c9-5c26f6f208e3\") " pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"
Apr 22 20:47:57.702443 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.702404    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6f31679e-6608-47b5-94c9-5c26f6f208e3-podres\") pod \"perf-node-gather-daemonset-wfpkc\" (UID: \"6f31679e-6608-47b5-94c9-5c26f6f208e3\") " pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"
Apr 22 20:47:57.702512 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.702470    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f31679e-6608-47b5-94c9-5c26f6f208e3-sys\") pod \"perf-node-gather-daemonset-wfpkc\" (UID: \"6f31679e-6608-47b5-94c9-5c26f6f208e3\") " pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"
Apr 22 20:47:57.702512 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.702506    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6f31679e-6608-47b5-94c9-5c26f6f208e3-proc\") pod \"perf-node-gather-daemonset-wfpkc\" (UID: \"6f31679e-6608-47b5-94c9-5c26f6f208e3\") " pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"
Apr 22 20:47:57.702618 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.702526    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f31679e-6608-47b5-94c9-5c26f6f208e3-lib-modules\") pod \"perf-node-gather-daemonset-wfpkc\" (UID: \"6f31679e-6608-47b5-94c9-5c26f6f208e3\") " pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"
Apr 22 20:47:57.702618 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.702569    2574 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c28x2\" (UniqueName: \"kubernetes.io/projected/6f31679e-6608-47b5-94c9-5c26f6f208e3-kube-api-access-c28x2\") pod \"perf-node-gather-daemonset-wfpkc\" (UID: \"6f31679e-6608-47b5-94c9-5c26f6f208e3\") " pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"
Apr 22 20:47:57.702618 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.702576    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6f31679e-6608-47b5-94c9-5c26f6f208e3-podres\") pod \"perf-node-gather-daemonset-wfpkc\" (UID: \"6f31679e-6608-47b5-94c9-5c26f6f208e3\") " pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"
Apr 22 20:47:57.702618 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.702606    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f31679e-6608-47b5-94c9-5c26f6f208e3-sys\") pod \"perf-node-gather-daemonset-wfpkc\" (UID: \"6f31679e-6608-47b5-94c9-5c26f6f208e3\") " pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"
Apr 22 20:47:57.702762 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.702615    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6f31679e-6608-47b5-94c9-5c26f6f208e3-proc\") pod \"perf-node-gather-daemonset-wfpkc\" (UID: \"6f31679e-6608-47b5-94c9-5c26f6f208e3\") " pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"
Apr 22 20:47:57.702762 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.702667    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f31679e-6608-47b5-94c9-5c26f6f208e3-lib-modules\") pod \"perf-node-gather-daemonset-wfpkc\" (UID: \"6f31679e-6608-47b5-94c9-5c26f6f208e3\") " pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"
Apr 22 20:47:57.709637 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.709616    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c28x2\" (UniqueName: \"kubernetes.io/projected/6f31679e-6608-47b5-94c9-5c26f6f208e3-kube-api-access-c28x2\") pod \"perf-node-gather-daemonset-wfpkc\" (UID: \"6f31679e-6608-47b5-94c9-5c26f6f208e3\") " pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"
Apr 22 20:47:57.822267 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.822211    2574 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"
Apr 22 20:47:57.944753 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:57.944726    2574 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"]
Apr 22 20:47:57.947172 ip-10-0-135-215 kubenswrapper[2574]: W0422 20:47:57.947124    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6f31679e_6608_47b5_94c9_5c26f6f208e3.slice/crio-6b91b9602a8c82bdf345762d33a48e01ecaf1ff56626247006910359f6a5aae2 WatchSource:0}: Error finding container 6b91b9602a8c82bdf345762d33a48e01ecaf1ff56626247006910359f6a5aae2: Status 404 returned error can't find the container with id 6b91b9602a8c82bdf345762d33a48e01ecaf1ff56626247006910359f6a5aae2
Apr 22 20:47:58.081003 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:58.080929    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fs69p_054ccdff-220c-47af-b943-4797d37b8723/dns/0.log"
Apr 22 20:47:58.102142 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:58.102124    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fs69p_054ccdff-220c-47af-b943-4797d37b8723/kube-rbac-proxy/0.log"
Apr 22 20:47:58.144020 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:58.144003    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sz29m_c4edbe2f-e978-4101-8d32-e35e7e5e740e/dns-node-resolver/0.log"
Apr 22 20:47:58.591133 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:58.591099    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vxzgj_a56d6e25-767b-43f8-b8e0-e1dd0b4879fa/node-ca/0.log"
Apr 22 20:47:58.785259 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:58.785220    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc" event={"ID":"6f31679e-6608-47b5-94c9-5c26f6f208e3","Type":"ContainerStarted","Data":"0e936aa481a4f7720105f00e432f73bd785068a9226384fba715fec841f55b61"}
Apr 22 20:47:58.785259 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:58.785252    2574 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc" event={"ID":"6f31679e-6608-47b5-94c9-5c26f6f208e3","Type":"ContainerStarted","Data":"6b91b9602a8c82bdf345762d33a48e01ecaf1ff56626247006910359f6a5aae2"}
Apr 22 20:47:58.785490 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:58.785337    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"
Apr 22 20:47:58.801776 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:58.801734    2574 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc" podStartSLOduration=1.8017198890000001 podStartE2EDuration="1.801719889s" podCreationTimestamp="2026-04-22 20:47:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-22 20:47:58.800406391 +0000 UTC m=+3030.218070043" watchObservedRunningTime="2026-04-22 20:47:58.801719889 +0000 UTC m=+3030.219383534"
Apr 22 20:47:59.557143 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:47:59.557104    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ct7n5_3a7698c5-bde7-4c7c-84ac-bcdba6d86851/serve-healthcheck-canary/0.log"
Apr 22 20:48:00.099639 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:00.099607    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kqn98_eb877699-2e26-41f4-8c95-f866532b38ae/kube-rbac-proxy/0.log"
Apr 22 20:48:00.118081 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:00.118051    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kqn98_eb877699-2e26-41f4-8c95-f866532b38ae/exporter/0.log"
Apr 22 20:48:00.137745 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:00.137720    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kqn98_eb877699-2e26-41f4-8c95-f866532b38ae/extractor/0.log"
Apr 22 20:48:02.000351 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:02.000319    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-68cc5db7c4-gj5c7_6de4e700-0c06-400b-845c-b372f8d40464/manager/0.log"
Apr 22 20:48:02.267007 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:02.266899    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-qs6b8_a8420c28-e439-4cc6-8d8b-3d960d64a87e/manager/0.log"
Apr 22 20:48:02.284925 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:02.284896    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-4snb2_68b9a15a-d6dc-4c5f-8d62-ead9a73bab96/s3-init/0.log"
Apr 22 20:48:04.800566 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:04.800527    2574 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xlhgw/perf-node-gather-daemonset-wfpkc"
Apr 22 20:48:05.665178 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:05.665142    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-74bb7799d9-pvlvw_45a1876b-06fe-44a6-ae0f-d15d8ad868c7/migrator/0.log"
Apr 22 20:48:05.684176 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:05.684140    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-74bb7799d9-pvlvw_45a1876b-06fe-44a6-ae0f-d15d8ad868c7/graceful-termination/0.log"
Apr 22 20:48:06.974172 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:06.974142    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7ttkd_84c4bd83-8750-4357-8974-fa2ff745d809/kube-multus/0.log"
Apr 22 20:48:07.001332 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:07.001309    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-js9v5_7f755d56-6d83-48c7-be8f-4efe2af27fbc/kube-multus-additional-cni-plugins/0.log"
Apr 22 20:48:07.020936 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:07.020911    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-js9v5_7f755d56-6d83-48c7-be8f-4efe2af27fbc/egress-router-binary-copy/0.log"
Apr 22 20:48:07.041619 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:07.041594    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-js9v5_7f755d56-6d83-48c7-be8f-4efe2af27fbc/cni-plugins/0.log"
Apr 22 20:48:07.064436 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:07.064392    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-js9v5_7f755d56-6d83-48c7-be8f-4efe2af27fbc/bond-cni-plugin/0.log"
Apr 22 20:48:07.087683 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:07.087656    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-js9v5_7f755d56-6d83-48c7-be8f-4efe2af27fbc/routeoverride-cni/0.log"
Apr 22 20:48:07.107881 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:07.107845    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-js9v5_7f755d56-6d83-48c7-be8f-4efe2af27fbc/whereabouts-cni-bincopy/0.log"
Apr 22 20:48:07.128349 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:07.128324    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-js9v5_7f755d56-6d83-48c7-be8f-4efe2af27fbc/whereabouts-cni/0.log"
Apr 22 20:48:07.491250 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:07.491202    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9cnw2_cdf7849a-206d-4b5a-ad81-e723c58f4426/network-metrics-daemon/0.log"
Apr 22 20:48:07.509118 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:07.509098    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9cnw2_cdf7849a-206d-4b5a-ad81-e723c58f4426/kube-rbac-proxy/0.log"
Apr 22 20:48:08.839429 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:08.839384    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlwx4_d54b3b0c-260b-4004-95d8-2d981031a8a0/ovn-controller/0.log"
Apr 22 20:48:08.868200 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:08.868107    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlwx4_d54b3b0c-260b-4004-95d8-2d981031a8a0/ovn-acl-logging/0.log"
Apr 22 20:48:08.887043 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:08.887023    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlwx4_d54b3b0c-260b-4004-95d8-2d981031a8a0/kube-rbac-proxy-node/0.log"
Apr 22 20:48:08.905681 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:08.905657    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlwx4_d54b3b0c-260b-4004-95d8-2d981031a8a0/kube-rbac-proxy-ovn-metrics/0.log"
Apr 22 20:48:08.922313 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:08.922293    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlwx4_d54b3b0c-260b-4004-95d8-2d981031a8a0/northd/0.log"
Apr 22 20:48:08.939713 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:08.939697    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlwx4_d54b3b0c-260b-4004-95d8-2d981031a8a0/nbdb/0.log"
Apr 22 20:48:08.958185 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:08.958168    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlwx4_d54b3b0c-260b-4004-95d8-2d981031a8a0/sbdb/0.log"
Apr 22 20:48:09.051245 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:09.051213    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zlwx4_d54b3b0c-260b-4004-95d8-2d981031a8a0/ovnkube-controller/0.log"
Apr 22 20:48:09.938488 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:09.938458    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-95q85_635d0f48-b24f-422c-b672-908590754a91/network-check-target-container/0.log"
Apr 22 20:48:10.832212 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:10.832182    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-8ht5v_2dca392c-613c-49fb-a976-970fe078c0c4/iptables-alerter/0.log"
Apr 22 20:48:11.421027 ip-10-0-135-215 kubenswrapper[2574]: I0422 20:48:11.420993    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-wgvqq_5063845e-1c48-48a7-bfa3-b31c3d6f8109/tuned/0.log"