Sep 10 15:42:01.077486 ip-10-0-128-145 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 15:42:01.479394 ip-10-0-128-145 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 15:42:01.479394 ip-10-0-128-145 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 15:42:01.479394 ip-10-0-128-145 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 15:42:01.479394 ip-10-0-128-145 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 15:42:01.479394 ip-10-0-128-145 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 15:42:01.480067 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.479469    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 15:42:01.483277 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483262    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:42:01.483277 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483278    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:42:01.483337 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483283    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:42:01.483337 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483286    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:42:01.483337 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483289    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:42:01.483337 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483292    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:42:01.483337 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483294    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:42:01.483337 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483297    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:42:01.483337 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483300    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:42:01.483337 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483302    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:42:01.483337 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483305    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:42:01.483337 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483307    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:42:01.483337 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483310    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:42:01.483337 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483313    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:42:01.483337 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483315    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:42:01.483337 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483318    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:42:01.483337 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483321    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:42:01.483337 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483324    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:42:01.483337 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483326    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:42:01.483337 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483329    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:42:01.483337 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483332    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:42:01.483337 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483340    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:42:01.483930 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483343    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:42:01.483930 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483345    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:42:01.483930 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483348    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:42:01.483930 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483351    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:42:01.483930 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483355    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:42:01.483930 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483358    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:42:01.483930 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483361    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:42:01.483930 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483372    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:42:01.483930 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483375    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:42:01.483930 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483377    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:42:01.483930 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483380    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:42:01.483930 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483382    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:42:01.483930 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483385    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:42:01.483930 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483387    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:42:01.483930 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483390    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:42:01.483930 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483393    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:42:01.483930 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483395    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:42:01.483930 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483397    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:42:01.483930 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483400    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:42:01.483930 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483402    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:42:01.484447 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483405    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:42:01.484447 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483407    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:42:01.484447 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483410    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:42:01.484447 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483412    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:42:01.484447 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483415    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:42:01.484447 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483417    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:42:01.484447 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483422    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:42:01.484447 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483426    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:42:01.484447 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483428    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:42:01.484447 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483431    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:42:01.484447 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483434    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:42:01.484447 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483437    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:42:01.484447 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483440    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:42:01.484447 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483444    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:42:01.484447 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483447    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:42:01.484447 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483450    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:42:01.484447 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483452    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:42:01.484447 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483455    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:42:01.484447 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483458    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:42:01.484910 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483461    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:42:01.484910 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483463    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:42:01.484910 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483466    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:42:01.484910 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483468    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:42:01.484910 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483471    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:42:01.484910 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483474    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:42:01.484910 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483476    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:42:01.484910 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483479    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:42:01.484910 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483481    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:42:01.484910 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483484    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:42:01.484910 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483487    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:42:01.484910 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483490    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:42:01.484910 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483492    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:42:01.484910 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483495    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:42:01.484910 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483498    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:42:01.484910 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483500    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:42:01.484910 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483503    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:42:01.484910 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483505    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:42:01.484910 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483508    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:42:01.485373 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483510    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:42:01.485373 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483513    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:42:01.485373 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483515    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:42:01.485373 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483518    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:42:01.485373 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483520    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:42:01.485373 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483523    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:42:01.485373 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483527    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:42:01.485373 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483531    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:42:01.485373 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483534    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:42:01.485373 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483536    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:42:01.485373 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483539    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:42:01.485373 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483542    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:42:01.485373 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483544    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:42:01.485373 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483547    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:42:01.485373 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483551    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:42:01.485373 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483554    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:42:01.485373 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483557    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:42:01.485373 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483559    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:42:01.485373 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.483562    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:42:01.485373 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485076    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:42:01.485844 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485084    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:42:01.485844 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485087    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:42:01.485844 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485090    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:42:01.485844 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485093    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:42:01.485844 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485096    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:42:01.485844 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485099    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:42:01.485844 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485101    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:42:01.485844 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485104    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:42:01.485844 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485107    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:42:01.485844 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485109    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:42:01.485844 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485112    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:42:01.485844 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485115    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:42:01.485844 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485117    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:42:01.485844 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485120    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:42:01.485844 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485122    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:42:01.485844 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485125    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:42:01.485844 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485128    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:42:01.485844 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485131    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:42:01.485844 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485133    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:42:01.485844 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485137    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:42:01.485844 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485140    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:42:01.486374 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485143    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:42:01.486374 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485145    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:42:01.486374 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485149    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:42:01.486374 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485153    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:42:01.486374 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485169    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:42:01.486374 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485174    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:42:01.486374 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485177    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:42:01.486374 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485181    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:42:01.486374 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485184    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:42:01.486374 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485186    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:42:01.486374 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485189    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:42:01.486374 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485191    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:42:01.486374 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485194    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:42:01.486374 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485196    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:42:01.486374 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485199    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:42:01.486374 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485202    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:42:01.486374 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485204    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:42:01.486374 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485207    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:42:01.486803 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485209    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:42:01.486803 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485212    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:42:01.486803 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485214    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:42:01.486803 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485217    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:42:01.486803 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485219    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:42:01.486803 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485222    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:42:01.486803 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485224    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:42:01.486803 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485227    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:42:01.486803 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485229    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:42:01.486803 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485232    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:42:01.486803 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485235    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:42:01.486803 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485237    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:42:01.486803 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485240    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:42:01.486803 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485242    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:42:01.486803 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485246    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:42:01.486803 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485249    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:42:01.486803 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485251    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:42:01.486803 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485254    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:42:01.486803 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485256    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:42:01.486803 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485259    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:42:01.487302 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485261    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:42:01.487302 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485264    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:42:01.487302 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485266    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:42:01.487302 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485269    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:42:01.487302 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485271    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:42:01.487302 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485274    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:42:01.487302 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485276    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:42:01.487302 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485279    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:42:01.487302 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485282    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:42:01.487302 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485286    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:42:01.487302 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485288    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:42:01.487302 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485291    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:42:01.487302 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485293    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:42:01.487302 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485296    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:42:01.487302 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485298    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:42:01.487302 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485301    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:42:01.487302 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485303    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:42:01.487302 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485306    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:42:01.487302 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485308    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:42:01.487787 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485311    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:42:01.487787 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485313    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:42:01.487787 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485316    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:42:01.487787 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485320    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:42:01.487787 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485323    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:42:01.487787 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485326    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:42:01.487787 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485328    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:42:01.487787 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485331    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:42:01.487787 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485334    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:42:01.487787 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485336    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:42:01.487787 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485338    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:42:01.487787 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485341    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:42:01.487787 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485343    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:42:01.487787 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485346    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:42:01.487787 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485348    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:42:01.487787 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485351    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:42:01.487787 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485353    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:42:01.487787 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485355    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:42:01.487787 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485358    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:42:01.487787 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485361    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:42:01.488292 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485731    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:42:01.488292 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485736    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:42:01.488292 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485739    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:42:01.488292 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485742    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:42:01.488292 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485745    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:42:01.488292 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485748    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:42:01.488292 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485751    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:42:01.488292 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485754    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:42:01.488292 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485757    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:42:01.488292 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485759    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:42:01.488292 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485762    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:42:01.488292 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485766    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:42:01.488292 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485770    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:42:01.488292 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485772    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:42:01.488292 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485775    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:42:01.488292 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485779    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:42:01.488292 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485782    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:42:01.488292 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485785    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:42:01.488292 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485788    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:42:01.488746 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485790    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:42:01.488746 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485793    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:42:01.488746 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485796    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:42:01.488746 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485798    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:42:01.488746 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485801    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:42:01.488746 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485803    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:42:01.488746 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485806    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:42:01.488746 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485808    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:42:01.488746 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485811    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:42:01.488746 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485813    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:42:01.488746 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485816    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:42:01.488746 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485818    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:42:01.488746 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485820    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:42:01.488746 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485823    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:42:01.488746 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485828    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:42:01.488746 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485831    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:42:01.488746 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485833    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:42:01.488746 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485835    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:42:01.488746 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485838    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:42:01.489247 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485841    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:42:01.489247 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485844    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:42:01.489247 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485848    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:42:01.489247 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485851    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:42:01.489247 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485854    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:42:01.489247 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485857    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:42:01.489247 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485860    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:42:01.489247 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485862    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:42:01.489247 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485866    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:42:01.489247 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485868    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:42:01.489247 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485871    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:42:01.489247 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485874    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:42:01.489247 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485876    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:42:01.489247 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485879    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:42:01.489247 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485882    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:42:01.489247 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485884    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:42:01.489247 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485887    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:42:01.489247 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485890    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:42:01.489247 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485892    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:42:01.489707 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485895    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:42:01.489707 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485899    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:42:01.489707 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485901    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:42:01.489707 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485904    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:42:01.489707 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485906    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:42:01.489707 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485909    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:42:01.489707 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485911    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:42:01.489707 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485913    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:42:01.489707 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485916    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:42:01.489707 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485918    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:42:01.489707 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485921    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:42:01.489707 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485923    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:42:01.489707 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485926    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:42:01.489707 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485928    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:42:01.489707 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485931    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:42:01.489707 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485933    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:42:01.489707 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485936    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:42:01.489707 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485939    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:42:01.489707 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485941    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:42:01.489707 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485944    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:42:01.490193 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485946    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:42:01.490193 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485949    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:42:01.490193 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485952    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:42:01.490193 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485954    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:42:01.490193 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485957    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:42:01.490193 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485959    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:42:01.490193 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485962    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:42:01.490193 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485964    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:42:01.490193 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485967    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:42:01.490193 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485970    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:42:01.490193 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485972    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:42:01.490193 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485974    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:42:01.490193 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485977    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:42:01.490193 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485979    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:42:01.490193 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485982    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:42:01.490193 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485984    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:42:01.490193 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485986    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:42:01.490193 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485989    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:42:01.490193 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485991    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:42:01.490193 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485994    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485996    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.485998    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486066    2564 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486099    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486105    2564 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486109    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486113    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486116    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486121    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486125    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486128    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486131    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486135    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486138    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486141    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486144    2564 flags.go:64] FLAG: --cgroup-root=""
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486148    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486150    2564 flags.go:64] FLAG: --client-ca-file=""
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486167    2564 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486171    2564 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486175    2564 flags.go:64] FLAG: --cluster-domain=""
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486178    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 15:42:01.490703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486182    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486185    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486188    2564 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486192    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486195    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486198    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486201    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486204    2564 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486209    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486211    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486215    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486218    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486221    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486233    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486251    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486273    2564 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486278    2564 flags.go:64] FLAG: --enable-server="true"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486282    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486291    2564 flags.go:64] FLAG: --event-burst="100"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486295    2564 flags.go:64] FLAG: --event-qps="50"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486299    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486302    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486305    2564 flags.go:64] FLAG: --eviction-hard=""
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486310    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 15:42:01.491255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486313    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486317    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486321    2564 flags.go:64] FLAG: --eviction-soft=""
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486325    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486328    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486331    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486334    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486337    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486340    2564 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486343    2564 flags.go:64] FLAG: --feature-gates=""
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486347    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486350    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486353    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486357    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486360    2564 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486363    2564 flags.go:64] FLAG: --help="false"
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486366    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-128-145.ec2.internal"
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486369    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486372    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486375    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486380    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486384    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486387    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 15:42:01.492063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486391    2564 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486394    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486397    2564 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486400    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486403    2564 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486406    2564 flags.go:64] FLAG: --kube-reserved=""
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486409    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486412    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486415    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486418    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486421    2564 flags.go:64] FLAG: --lock-file=""
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486423    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486426    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486430    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486435    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486438    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486441    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486443    2564 flags.go:64] FLAG: --logging-format="text"
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486446    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486449    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486452    2564 flags.go:64] FLAG: --manifest-url=""
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486455    2564 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486459    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486462    2564 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486467    2564 flags.go:64] FLAG: --max-pods="110"
Sep 10 15:42:01.492795 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486470    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486473    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486475    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486478    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486481    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486484    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486487    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486498    2564 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486502    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486505    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486508    2564 flags.go:64] FLAG: --pod-cidr=""
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486511    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486516    2564 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486519    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486522    2564 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486525    2564 flags.go:64] FLAG: --port="10250"
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486529    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486532    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b1a862289b9fd784"
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486535    2564 flags.go:64] FLAG: --qos-reserved=""
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486538    2564 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486540    2564 flags.go:64] FLAG: --register-node="true"
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486544    2564 flags.go:64] FLAG: --register-with-taints=""
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486547    2564 flags.go:64] FLAG: --registry-burst="10"
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486550    2564 flags.go:64] FLAG: --registry-qps="5"
Sep 10 15:42:01.493425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486553    2564 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486556    2564 flags.go:64] FLAG: --reserved-memory=""
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486559    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486562    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486565    2564 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486568    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486570    2564 flags.go:64] FLAG: --runonce="false"
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486573    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486576    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486579    2564 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486582    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486584    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486587    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486590    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486593    2564 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486597    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486601    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486604    2564 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486606    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486609    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486612    2564 flags.go:64] FLAG: --system-cgroups=""
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486615    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486621    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486623    2564 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486626    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 15:42:01.494031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486631    2564 flags.go:64] FLAG: --tls-min-version=""
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486633    2564 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486636    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486639    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486642    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486645    2564 flags.go:64] FLAG: --v="2"
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486655    2564 flags.go:64] FLAG: --version="false"
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486659    2564 flags.go:64] FLAG: --vmodule=""
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486663    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.486666    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486790    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486795    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486798    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486801    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486804    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486807    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486810    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486813    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486816    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486818    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486821    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486823    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:42:01.494770 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486826    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:42:01.495326 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486831    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:42:01.495326 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486834    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:42:01.495326 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486836    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:42:01.495326 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486839    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:42:01.495326 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486841    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:42:01.495326 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486844    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:42:01.495326 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486846    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:42:01.495326 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486850    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:42:01.495326 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486852    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:42:01.495326 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486855    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:42:01.495326 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486857    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:42:01.495326 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486861    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:42:01.495326 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486863    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:42:01.495326 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486866    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:42:01.495326 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486869    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:42:01.495326 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486872    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:42:01.495326 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486875    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:42:01.495326 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486877    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:42:01.495326 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486880    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:42:01.495326 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486882    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:42:01.495823 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486885    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:42:01.495823 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486887    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:42:01.495823 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486890    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:42:01.495823 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486893    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:42:01.495823 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486895    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:42:01.495823 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486898    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:42:01.495823 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486900    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:42:01.495823 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486903    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:42:01.495823 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486905    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:42:01.495823 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486907    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:42:01.495823 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486910    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:42:01.495823 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486912    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:42:01.495823 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486917    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:42:01.495823 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486919    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:42:01.495823 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486922    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:42:01.495823 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486924    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:42:01.495823 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486927    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:42:01.495823 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486929    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:42:01.495823 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486932    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:42:01.495823 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486935    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:42:01.504150 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486937    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:42:01.504150 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486941    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:42:01.504150 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486945    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:42:01.504150 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486948    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:42:01.504150 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486950    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:42:01.504150 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486953    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:42:01.504150 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486955    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:42:01.504150 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486958    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:42:01.504150 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486961    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:42:01.504150 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486964    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:42:01.504150 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486967    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:42:01.504150 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486969    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:42:01.504150 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486972    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:42:01.504150 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486974    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:42:01.504150 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486977    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:42:01.504150 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486979    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:42:01.504150 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486982    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:42:01.504150 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486984    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:42:01.504150 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486987    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:42:01.504617 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486989    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:42:01.504617 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486991    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:42:01.504617 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486995    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:42:01.504617 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.486999    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:42:01.504617 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487002    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:42:01.504617 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487006    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:42:01.504617 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487009    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:42:01.504617 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487012    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:42:01.504617 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487016    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:42:01.504617 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487019    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:42:01.504617 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487021    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:42:01.504617 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487024    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:42:01.504617 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487026    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:42:01.504617 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487029    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:42:01.504617 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487031    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:42:01.504617 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487034    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:42:01.504617 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487036    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:42:01.504617 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487039    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:42:01.504617 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487041    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:42:01.505084 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487044    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:42:01.505084 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487046    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:42:01.505084 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487049    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:42:01.505084 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487052    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:42:01.505084 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487054    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:42:01.505084 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487057    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:42:01.505084 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487059    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:42:01.505084 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.487062    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:42:01.505084 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.487663    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 15:42:01.505084 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.495321    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 15:42:01.505084 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.495340    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 15:42:01.505084 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.496241    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 15:42:01.505084 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.496249    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 15:42:01.505084 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.496776    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:42:01.505084 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.496866    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:42:01.505514 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.496927    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:42:01.505514 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497027    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:42:01.505514 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497030    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:42:01.505514 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497033    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:42:01.505514 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497036    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:42:01.505514 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497039    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:42:01.505514 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497042    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:42:01.505514 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497045    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:42:01.505514 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497049    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:42:01.505514 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497219    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:42:01.505514 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497225    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:42:01.505514 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497229    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:42:01.505514 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497231    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:42:01.505514 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497234    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:42:01.505514 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497237    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:42:01.505514 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497239    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:42:01.505514 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497242    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:42:01.505514 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497245    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:42:01.505514 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497247    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:42:01.505973 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497250    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:42:01.505973 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497252    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:42:01.505973 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497255    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:42:01.505973 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497258    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:42:01.505973 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497260    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:42:01.505973 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497263    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:42:01.505973 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497271    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:42:01.505973 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497274    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:42:01.505973 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497278    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:42:01.505973 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497281    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:42:01.505973 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497283    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:42:01.505973 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497286    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:42:01.505973 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497289    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:42:01.505973 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497292    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:42:01.505973 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497294    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:42:01.505973 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497297    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:42:01.505973 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497299    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:42:01.505973 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497302    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:42:01.505973 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497304    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:42:01.506441 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497307    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:42:01.506441 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497311    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:42:01.506441 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497316    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:42:01.506441 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497319    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:42:01.506441 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497321    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:42:01.506441 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497324    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:42:01.506441 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497327    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:42:01.506441 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497330    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:42:01.506441 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497332    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:42:01.506441 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497335    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:42:01.506441 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497337    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:42:01.506441 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497341    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:42:01.506441 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497345    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:42:01.506441 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497348    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:42:01.506441 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497351    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:42:01.506441 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497354    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:42:01.506441 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497357    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:42:01.506441 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497359    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:42:01.506441 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497362    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:42:01.506906 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497364    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:42:01.506906 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497367    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:42:01.506906 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497369    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:42:01.506906 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497372    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:42:01.506906 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497374    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:42:01.506906 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497377    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:42:01.506906 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497380    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:42:01.506906 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497382    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:42:01.506906 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497385    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:42:01.506906 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497387    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:42:01.506906 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497390    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:42:01.506906 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497392    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:42:01.506906 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497395    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:42:01.506906 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497398    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:42:01.506906 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497400    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:42:01.506906 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497403    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:42:01.506906 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497405    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:42:01.506906 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497408    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:42:01.506906 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497410    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:42:01.506906 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497413    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:42:01.507468 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497416    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:42:01.507468 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497419    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:42:01.507468 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497421    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:42:01.507468 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497424    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:42:01.507468 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497426    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:42:01.507468 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497429    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:42:01.507468 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497432    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:42:01.507468 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497435    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:42:01.507468 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497438    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:42:01.507468 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497441    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:42:01.507468 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497444    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:42:01.507468 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497446    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:42:01.507468 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497449    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:42:01.507468 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497451    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:42:01.507468 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497454    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:42:01.507468 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497457    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:42:01.507468 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497459    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:42:01.507468 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497461    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:42:01.507468 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497464    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:42:01.507468 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497466    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:42:01.507941 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.497471    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 15:42:01.507941 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497599    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:42:01.507941 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497605    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:42:01.507941 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497609    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:42:01.507941 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497612    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:42:01.507941 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497614    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:42:01.507941 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497617    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:42:01.507941 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497620    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:42:01.507941 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497622    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:42:01.507941 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497624    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:42:01.507941 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497627    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:42:01.507941 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497629    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:42:01.507941 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497632    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:42:01.507941 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497635    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:42:01.507941 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497638    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:42:01.508350 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497641    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:42:01.508350 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497643    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:42:01.508350 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497645    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:42:01.508350 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497648    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:42:01.508350 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497651    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:42:01.508350 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497653    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:42:01.508350 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497656    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:42:01.508350 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497658    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:42:01.508350 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497661    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:42:01.508350 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497663    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:42:01.508350 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497666    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:42:01.508350 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497669    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:42:01.508350 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497671    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:42:01.508350 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497674    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:42:01.508350 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497676    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:42:01.508350 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497679    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:42:01.508350 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497681    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:42:01.508350 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497684    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:42:01.508350 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497686    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:42:01.508350 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497688    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:42:01.508832 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497691    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:42:01.508832 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497693    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:42:01.508832 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497696    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:42:01.508832 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497698    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:42:01.508832 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497701    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:42:01.508832 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497703    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:42:01.508832 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497705    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:42:01.508832 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497708    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:42:01.508832 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497710    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:42:01.508832 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497714    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:42:01.508832 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497718    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:42:01.508832 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497721    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:42:01.508832 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497724    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:42:01.508832 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497726    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:42:01.508832 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497729    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:42:01.508832 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497731    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:42:01.508832 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497733    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:42:01.508832 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497736    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:42:01.508832 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497738    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:42:01.508832 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497740    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:42:01.509347 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497744    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:42:01.509347 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497748    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:42:01.509347 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497751    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:42:01.509347 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497753    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:42:01.509347 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497756    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:42:01.509347 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497758    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:42:01.509347 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497761    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:42:01.509347 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497763    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:42:01.509347 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497766    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:42:01.509347 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497768    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:42:01.509347 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497771    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:42:01.509347 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497773    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:42:01.509347 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497775    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:42:01.509347 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497778    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:42:01.509347 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497780    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:42:01.509347 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497783    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:42:01.509347 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497785    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:42:01.509347 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497787    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:42:01.509347 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497790    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:42:01.509347 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497792    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:42:01.509810 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497795    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:42:01.509810 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497797    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:42:01.509810 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497800    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:42:01.509810 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497802    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:42:01.509810 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497806    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:42:01.509810 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497808    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:42:01.509810 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497811    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:42:01.509810 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497813    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:42:01.509810 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497815    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:42:01.509810 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497818    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:42:01.509810 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497820    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:42:01.509810 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497823    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:42:01.509810 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497825    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:42:01.509810 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497829    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:42:01.509810 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497835    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:42:01.509810 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497838    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:42:01.509810 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497840    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:42:01.509810 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497843    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:42:01.509810 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497846    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:42:01.510294 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497849    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:42:01.510294 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497851    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:42:01.510294 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497854    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:42:01.510294 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497857    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:42:01.510294 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497859    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:42:01.510294 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:01.497862    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:42:01.510294 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.497866    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 15:42:01.510294 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.497963    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 15:42:01.510294 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.501001    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 15:42:01.510294 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.501999    2564 server.go:1019] "Starting client certificate rotation"
Sep 10 15:42:01.510294 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.502102    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 15:42:01.510294 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.502132    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 15:42:01.510294 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.504400    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 15:42:01.510623 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.508736    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 15:42:01.517140 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.517109    2564 log.go:25] "Validated CRI v1 runtime API"
Sep 10 15:42:01.519755 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.519739    2564 log.go:25] "Validated CRI v1 image API"
Sep 10 15:42:01.520564 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.520550    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 15:42:01.523965 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.523944    2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b4b8fe63-fb1e-47d5-9ac1-19dd07e08f0d:/dev/nvme0n1p4 b510e391-40c7-4637-a6a7-3ae7fbc072db:/dev/nvme0n1p3]
Sep 10 15:42:01.524036 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.523964    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 15:42:01.530289 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.530171    2564 manager.go:216] Machine: {Timestamp:2026-09-10 15:42:01.528959806 +0000 UTC m=+0.333279057 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3201328 MemoryCapacity:32812146688 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2db45beda1b14aeb9924b82f58834b SystemUUID:ec2db45b-eda1-b14a-eb99-24b82f58834b BootID:43eed0d0-a73c-43dc-86a4-88ff0302d8db Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406073344 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:54:da:13:97:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:54:da:13:97:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:2f:18:15:7f:4c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812146688 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 15:42:01.530289 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.530277    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 15:42:01.530450 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.530384    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 15:42:01.530920 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.530900    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 15:42:01.533256 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.533230    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 15:42:01.533391 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.533259    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-145.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 15:42:01.534147 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.534137    2564 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 15:42:01.534196 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.534150    2564 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 15:42:01.534196 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.534175    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 15:42:01.535114 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.535105    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 15:42:01.536539 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.536529    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 15:42:01.539822 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.539811    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 15:42:01.539935 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.539927    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 15:42:01.539969 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.539961    2564 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 15:42:01.539997 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.539970    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 15:42:01.539997 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.539984    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 15:42:01.539997 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.539994    2564 kubelet.go:398] "Adding apiserver pod source"
Sep 10 15:42:01.540089 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.540002    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 15:42:01.540384 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.540365    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 15:42:01.540481 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.540390    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 15:42:01.542660 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.542644    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 15:42:01.545032 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.545017    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 15:42:01.545108 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.545056    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 15:42:01.545839 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.545827    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 15:42:01.545886 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.545844    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 15:42:01.545886 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.545851    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 15:42:01.545886 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.545857    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 15:42:01.545886 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.545865    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 15:42:01.545886 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.545870    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 15:42:01.545886 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.545883    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 15:42:01.546036 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.545890    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 15:42:01.546036 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.545897    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 15:42:01.546036 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.545902    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 15:42:01.546036 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.545915    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 15:42:01.546036 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.545920    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 15:42:01.546575 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.546564    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 15:42:01.546611 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.546577    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 15:42:01.550143 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.550124    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-145.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 15:42:01.550385 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.550365    2564 server.go:1268] "Started kubelet"
Sep 10 15:42:01.550487 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.550423    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 15:42:01.550610 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.550569    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 15:42:01.550658 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.550628    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 15:42:01.550905 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:01.550874    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 15:42:01.550905 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:01.550874    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-145.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 15:42:01.551240 ip-10-0-128-145 systemd[1]: Started Kubernetes Kubelet.
Sep 10 15:42:01.551907 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.551891    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 15:42:01.552220 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.552185    2564 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 15:42:01.556008 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.555985    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 15:42:01.556577 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.556560    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 15:42:01.556703 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:01.556681    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-145.ec2.internal\" not found"
Sep 10 15:42:01.557704 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.557540    2564 factory.go:55] Registering systemd factory
Sep 10 15:42:01.557791 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.557710    2564 factory.go:223] Registration of the systemd container factory successfully
Sep 10 15:42:01.557993 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.557965    2564 factory.go:153] Registering CRI-O factory
Sep 10 15:42:01.557993 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.557988    2564 factory.go:223] Registration of the crio container factory successfully
Sep 10 15:42:01.558103 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.558058    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 15:42:01.558103 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.558082    2564 factory.go:103] Registering Raw factory
Sep 10 15:42:01.558222 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.558103    2564 manager.go:1196] Started watching for new ooms in manager
Sep 10 15:42:01.558437 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.558417    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 15:42:01.558437 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.558423    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 15:42:01.558573 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.558447    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 15:42:01.558573 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:01.558519    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 15:42:01.558573 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.558539    2564 manager.go:349] Starting recovery of all containers
Sep 10 15:42:01.558573 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.558544    2564 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 15:42:01.558573 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.558551    2564 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 15:42:01.559717 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:01.559687    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-145.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 15:42:01.562776 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:01.559802    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-145.ec2.internal.18d3fff8eaac0882  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-145.ec2.internal,UID:ip-10-0-128-145.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-145.ec2.internal,},FirstTimestamp:2026-09-10 15:42:01.550334082 +0000 UTC m=+0.354653334,LastTimestamp:2026-09-10 15:42:01.550334082 +0000 UTC m=+0.354653334,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-145.ec2.internal,}"
Sep 10 15:42:01.567922 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.567883    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jr6jx"
Sep 10 15:42:01.568019 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:01.567940    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 15:42:01.572353 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.572338    2564 manager.go:354] Recovery completed
Sep 10 15:42:01.574001 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.573986    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jr6jx"
Sep 10 15:42:01.576463 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.576449    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:01.578198 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.578181    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:01.578269 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.578219    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:01.578269 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.578233    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:01.578697 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.578684    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 15:42:01.578697 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.578695    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 15:42:01.578697 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.578696    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:01.578832 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.578712    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 15:42:01.579409 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.579393    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:01.579478 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.579427    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:01.579478 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.579442    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:01.579932 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:01.579862    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-145.ec2.internal.18d3fff8ec553b77  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-145.ec2.internal,UID:ip-10-0-128-145.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-145.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-145.ec2.internal,},FirstTimestamp:2026-09-10 15:42:01.578199927 +0000 UTC m=+0.382519378,LastTimestamp:2026-09-10 15:42:01.578199927 +0000 UTC m=+0.382519378,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-145.ec2.internal,}"
Sep 10 15:42:01.580008 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.579974    2564 policy_none.go:49] "None policy: Start"
Sep 10 15:42:01.580008 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.579987    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 15:42:01.580008 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.579997    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 15:42:01.581042 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.581031    2564 policy_none.go:47] "Start"
Sep 10 15:42:01.628718 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.628702    2564 manager.go:337] "Starting Device Plugin manager"
Sep 10 15:42:01.629815 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:01.628740    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 15:42:01.629815 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.628750    2564 server.go:87] "Starting device plugin registration server"
Sep 10 15:42:01.629815 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.629095    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 15:42:01.629815 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.629109    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 15:42:01.629815 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.629232    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 15:42:01.629815 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.629314    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 15:42:01.629815 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.629323    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 15:42:01.630036 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:01.629839    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 15:42:01.630036 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:01.629891    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-145.ec2.internal\" not found"
Sep 10 15:42:01.664251 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.664214    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 15:42:01.665525 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.665509    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 15:42:01.665586 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.665555    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 15:42:01.665586 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.665578    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 15:42:01.665674 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:01.665611    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 15:42:01.667418 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.667403    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:42:01.729865 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.729776    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:01.730796 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.730778    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:01.730883 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.730809    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:01.730883 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.730826    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:01.730883 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.730854    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-145.ec2.internal"
Sep 10 15:42:01.738845 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.738832    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-145.ec2.internal"
Sep 10 15:42:01.738896 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:01.738852    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-145.ec2.internal\": node \"ip-10-0-128-145.ec2.internal\" not found"
Sep 10 15:42:01.752598 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:01.752580    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-145.ec2.internal\" not found"
Sep 10 15:42:01.765854 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.765832    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-145.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-145.ec2.internal"]
Sep 10 15:42:01.765951 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.765895    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:01.767276 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.767257    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:01.767378 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.767286    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:01.767378 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.767296    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:01.768570 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.768557    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:01.768692 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.768667    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-145.ec2.internal"
Sep 10 15:42:01.768734 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.768706    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:01.769277 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.769260    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:01.769277 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.769275    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:01.769413 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.769290    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:01.769413 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.769297    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:01.769413 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.769311    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:01.769413 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.769300    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:01.770615 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.770598    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-145.ec2.internal"
Sep 10 15:42:01.770691 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.770623    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:01.771230 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.771215    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:01.771317 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.771244    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:01.771317 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.771256    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:01.788697 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:01.788678    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-145.ec2.internal\" not found" node="ip-10-0-128-145.ec2.internal"
Sep 10 15:42:01.793276 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:01.793256    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-145.ec2.internal\" not found" node="ip-10-0-128-145.ec2.internal"
Sep 10 15:42:01.853305 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:01.853279    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-145.ec2.internal\" not found"
Sep 10 15:42:01.860976 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.860955    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8dcad2a033892fede0cdda710bab032c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-145.ec2.internal\" (UID: \"8dcad2a033892fede0cdda710bab032c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-145.ec2.internal"
Sep 10 15:42:01.861044 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.860982    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dcad2a033892fede0cdda710bab032c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-145.ec2.internal\" (UID: \"8dcad2a033892fede0cdda710bab032c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-145.ec2.internal"
Sep 10 15:42:01.861044 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.860998    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05d404cb45a3a313356b08f70399e53d-config\") pod \"kube-apiserver-proxy-ip-10-0-128-145.ec2.internal\" (UID: \"05d404cb45a3a313356b08f70399e53d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-145.ec2.internal"
Sep 10 15:42:01.953592 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:01.953553    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-145.ec2.internal\" not found"
Sep 10 15:42:01.961958 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.961942    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8dcad2a033892fede0cdda710bab032c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-145.ec2.internal\" (UID: \"8dcad2a033892fede0cdda710bab032c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-145.ec2.internal"
Sep 10 15:42:01.962010 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.961964    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dcad2a033892fede0cdda710bab032c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-145.ec2.internal\" (UID: \"8dcad2a033892fede0cdda710bab032c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-145.ec2.internal"
Sep 10 15:42:01.962010 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.961981    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05d404cb45a3a313356b08f70399e53d-config\") pod \"kube-apiserver-proxy-ip-10-0-128-145.ec2.internal\" (UID: \"05d404cb45a3a313356b08f70399e53d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-145.ec2.internal"
Sep 10 15:42:01.962090 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.962008    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05d404cb45a3a313356b08f70399e53d-config\") pod \"kube-apiserver-proxy-ip-10-0-128-145.ec2.internal\" (UID: \"05d404cb45a3a313356b08f70399e53d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-145.ec2.internal"
Sep 10 15:42:01.962090 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.962035    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8dcad2a033892fede0cdda710bab032c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-145.ec2.internal\" (UID: \"8dcad2a033892fede0cdda710bab032c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-145.ec2.internal"
Sep 10 15:42:01.962090 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:01.962047    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dcad2a033892fede0cdda710bab032c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-145.ec2.internal\" (UID: \"8dcad2a033892fede0cdda710bab032c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-145.ec2.internal"
Sep 10 15:42:02.054470 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:02.054389    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-145.ec2.internal\" not found"
Sep 10 15:42:02.091912 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:02.091887    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-145.ec2.internal"
Sep 10 15:42:02.095895 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:02.095860    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-145.ec2.internal"
Sep 10 15:42:02.154598 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:02.154559    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-145.ec2.internal\" not found"
Sep 10 15:42:02.255541 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:02.255509    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-145.ec2.internal\" not found"
Sep 10 15:42:02.356128 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:02.356047    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-145.ec2.internal\" not found"
Sep 10 15:42:02.456646 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:02.456614    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-145.ec2.internal\" not found"
Sep 10 15:42:02.502181 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:02.502141    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 15:42:02.502589 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:02.502313    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 15:42:02.527681 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:02.527654    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:42:02.556851 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:02.556826    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-145.ec2.internal\" not found"
Sep 10 15:42:02.576456 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:02.576419    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 15:37:01 +0000 UTC" deadline="2028-05-15 04:04:10.053096245 +0000 UTC"
Sep 10 15:42:02.576456 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:02.576453    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14700h22m7.4766465s"
Sep 10 15:42:02.636081 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:02.636051    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d404cb45a3a313356b08f70399e53d.slice/crio-ee37123828ba7457401dc9aeb294d1f50f726140e82b636b91e2cbb69c85828f WatchSource:0}: Error finding container ee37123828ba7457401dc9aeb294d1f50f726140e82b636b91e2cbb69c85828f: Status 404 returned error can't find the container with id ee37123828ba7457401dc9aeb294d1f50f726140e82b636b91e2cbb69c85828f
Sep 10 15:42:02.641034 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:02.641016    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:42:02.657225 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:02.657200    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-145.ec2.internal\" not found"
Sep 10 15:42:02.668700 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:02.668659    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-145.ec2.internal" event={"ID":"05d404cb45a3a313356b08f70399e53d","Type":"ContainerStarted","Data":"ee37123828ba7457401dc9aeb294d1f50f726140e82b636b91e2cbb69c85828f"}
Sep 10 15:42:02.757919 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:02.757885    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-145.ec2.internal\" not found"
Sep 10 15:42:02.849782 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:02.849755    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:42:02.857785 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:02.857764    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-145.ec2.internal"
Sep 10 15:42:02.868595 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:02.868573    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 15:42:02.869619 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:02.869606    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-145.ec2.internal"
Sep 10 15:42:02.875837 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:02.875816    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 15:42:02.889991 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:02.889931    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:42:03.484034 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.483998    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:42:03.541152 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.541107    2564 apiserver.go:52] "Watching apiserver"
Sep 10 15:42:03.550758 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.550729    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 15:42:03.551163 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.551139    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-66dx8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-145.ec2.internal","openshift-multus/multus-additional-cni-plugins-bjjkl","openshift-multus/multus-pl75t","openshift-multus/network-metrics-daemon-m8bvz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh","openshift-image-registry/node-ca-k6fhq","openshift-network-diagnostics/network-check-target-g5l6v","openshift-network-operator/iptables-alerter-k5bnn","openshift-ovn-kubernetes/ovnkube-node-4xzlm","kube-system/global-pull-secret-syncer-znbph","kube-system/konnectivity-agent-q7c8n","kube-system/kube-apiserver-proxy-ip-10-0-128-145.ec2.internal"]
Sep 10 15:42:03.552853 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.552803    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k6fhq"
Sep 10 15:42:03.554199 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.554161    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.555486 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.555463    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 15:42:03.555598 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.555490    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.555671 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.555601    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 15:42:03.555671 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.555469    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vsspx\""
Sep 10 15:42:03.555820 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.555801    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 15:42:03.556636 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.556618    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:03.556761 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:03.556705    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8bvz" podUID="a140ffcb-b59c-4e55-95e8-e9ed589396b1"
Sep 10 15:42:03.557145 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.557120    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 15:42:03.557413 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.557396    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lj8xg\""
Sep 10 15:42:03.557671 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.557654    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 15:42:03.557892 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.557877    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 15:42:03.558075 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.558034    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.558075 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.558046    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 15:42:03.558075 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.558060    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 15:42:03.558327 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.558097    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 15:42:03.559498 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.558933    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.559498 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.558976    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 15:42:03.559498 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.559360    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nj7mv\""
Sep 10 15:42:03.560299 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.560282    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:03.560514 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.560424    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 15:42:03.560514 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:03.560425    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5l6v" podUID="f5ced900-9635-4cf4-9390-7396c3ef86a7"
Sep 10 15:42:03.560715 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.560697    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5tfsg\""
Sep 10 15:42:03.561057 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.561037    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 15:42:03.561149 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.561075    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 15:42:03.562769 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.562071    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:42:03.562769 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.562087    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f2s6q\""
Sep 10 15:42:03.562769 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.562140    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 15:42:03.562933 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.562816    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k5bnn"
Sep 10 15:42:03.565706 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.565683    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wm8km\""
Sep 10 15:42:03.566146 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.566127    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 15:42:03.566391 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.566375    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 15:42:03.566475 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.566422    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:42:03.566798 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.566777    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.568269 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.568250    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:03.568381 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:03.568337    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-znbph" podUID="8f29f7a6-d451-487d-92f1-84c47bf32d5c"
Sep 10 15:42:03.569481 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.569461    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q7c8n"
Sep 10 15:42:03.569635 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.569615    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4e0052e2-036e-47d2-9881-5d69873ddff4-sys-fs\") pod \"aws-ebs-csi-driver-node-sglsh\" (UID: \"4e0052e2-036e-47d2-9881-5d69873ddff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.569694 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.569651    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-etc-modprobe-d\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.569694 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.569688    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e6f2f68d-170c-4567-ba06-b05bea2cb642-serviceca\") pod \"node-ca-k6fhq\" (UID: \"e6f2f68d-170c-4567-ba06-b05bea2cb642\") " pod="openshift-image-registry/node-ca-k6fhq"
Sep 10 15:42:03.569793 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.569713    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-multus-cni-dir\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.569793 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.569740    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-host-run-k8s-cni-cncf-io\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.569793 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.569765    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-host-var-lib-cni-multus\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.569793 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.569789    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4e0052e2-036e-47d2-9881-5d69873ddff4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sglsh\" (UID: \"4e0052e2-036e-47d2-9881-5d69873ddff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.569956 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.569813    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4e0052e2-036e-47d2-9881-5d69873ddff4-registration-dir\") pod \"aws-ebs-csi-driver-node-sglsh\" (UID: \"4e0052e2-036e-47d2-9881-5d69873ddff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.569956 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.569839    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-etc-systemd\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.569956 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.569861    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-sys\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.569956 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.569886    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1601814-5e66-4f97-8165-920af19fb9a4-cni-binary-copy\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.569956 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.569910    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-multus-socket-dir-parent\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.569956 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.569935    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmvbb\" (UniqueName: \"kubernetes.io/projected/e1601814-5e66-4f97-8165-920af19fb9a4-kube-api-access-fmvbb\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.570209 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.569962    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-etc-sysconfig\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.570209 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.569986    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-var-lib-kubelet\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.570209 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570026    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9cde9140-e403-43b0-bbda-d78cd59ce6d1-etc-tuned\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.570209 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570096    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e30c4619-7a14-46c2-b2e8-f18c219cfa66-cni-binary-copy\") pod \"multus-additional-cni-plugins-bjjkl\" (UID: \"e30c4619-7a14-46c2-b2e8-f18c219cfa66\") " pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.570209 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570131    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e30c4619-7a14-46c2-b2e8-f18c219cfa66-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bjjkl\" (UID: \"e30c4619-7a14-46c2-b2e8-f18c219cfa66\") " pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.570209 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570156    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs\") pod \"network-metrics-daemon-m8bvz\" (UID: \"a140ffcb-b59c-4e55-95e8-e9ed589396b1\") " pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:03.570209 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570185    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzxwl\" (UniqueName: \"kubernetes.io/projected/4e0052e2-036e-47d2-9881-5d69873ddff4-kube-api-access-qzxwl\") pod \"aws-ebs-csi-driver-node-sglsh\" (UID: \"4e0052e2-036e-47d2-9881-5d69873ddff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.570543 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570209    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-etc-sysctl-conf\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.570543 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570294    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e30c4619-7a14-46c2-b2e8-f18c219cfa66-system-cni-dir\") pod \"multus-additional-cni-plugins-bjjkl\" (UID: \"e30c4619-7a14-46c2-b2e8-f18c219cfa66\") " pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.570543 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570319    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-cnibin\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.570543 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570368    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-host-var-lib-kubelet\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.570543 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570397    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-hostroot\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.570543 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570414    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4e0052e2-036e-47d2-9881-5d69873ddff4-socket-dir\") pod \"aws-ebs-csi-driver-node-sglsh\" (UID: \"4e0052e2-036e-47d2-9881-5d69873ddff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.570543 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570428    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-run\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.570543 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570443    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-host\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.570543 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570462    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cdtx\" (UniqueName: \"kubernetes.io/projected/e6f2f68d-170c-4567-ba06-b05bea2cb642-kube-api-access-2cdtx\") pod \"node-ca-k6fhq\" (UID: \"e6f2f68d-170c-4567-ba06-b05bea2cb642\") " pod="openshift-image-registry/node-ca-k6fhq"
Sep 10 15:42:03.570543 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570497    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-multus-conf-dir\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.570543 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570513    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1601814-5e66-4f97-8165-920af19fb9a4-multus-daemon-config\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.570543 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570536    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4e0052e2-036e-47d2-9881-5d69873ddff4-etc-selinux\") pod \"aws-ebs-csi-driver-node-sglsh\" (UID: \"4e0052e2-036e-47d2-9881-5d69873ddff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.571308 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570578    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skmlt\" (UniqueName: \"kubernetes.io/projected/9cde9140-e403-43b0-bbda-d78cd59ce6d1-kube-api-access-skmlt\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.571308 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570603    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6f2f68d-170c-4567-ba06-b05bea2cb642-host\") pod \"node-ca-k6fhq\" (UID: \"e6f2f68d-170c-4567-ba06-b05bea2cb642\") " pod="openshift-image-registry/node-ca-k6fhq"
Sep 10 15:42:03.571308 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570644    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e30c4619-7a14-46c2-b2e8-f18c219cfa66-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bjjkl\" (UID: \"e30c4619-7a14-46c2-b2e8-f18c219cfa66\") " pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.571308 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570704    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d465t\" (UniqueName: \"kubernetes.io/projected/e30c4619-7a14-46c2-b2e8-f18c219cfa66-kube-api-access-d465t\") pod \"multus-additional-cni-plugins-bjjkl\" (UID: \"e30c4619-7a14-46c2-b2e8-f18c219cfa66\") " pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.571308 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570739    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-etc-kubernetes\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.571308 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570764    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9cde9140-e403-43b0-bbda-d78cd59ce6d1-tmp\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.571308 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570789    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4e0052e2-036e-47d2-9881-5d69873ddff4-device-dir\") pod \"aws-ebs-csi-driver-node-sglsh\" (UID: \"4e0052e2-036e-47d2-9881-5d69873ddff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.571308 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570823    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-etc-kubernetes\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.571308 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570846    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-etc-sysctl-d\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.571308 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570868    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-lib-modules\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.571308 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570893    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e30c4619-7a14-46c2-b2e8-f18c219cfa66-cnibin\") pod \"multus-additional-cni-plugins-bjjkl\" (UID: \"e30c4619-7a14-46c2-b2e8-f18c219cfa66\") " pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.571308 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570913    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 15:42:03.571308 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570918    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-system-cni-dir\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.571308 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570924    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 15:42:03.571308 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570944    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-host-run-netns\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.571308 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570946    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s29k5\""
Sep 10 15:42:03.571308 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.570972    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e30c4619-7a14-46c2-b2e8-f18c219cfa66-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bjjkl\" (UID: \"e30c4619-7a14-46c2-b2e8-f18c219cfa66\") " pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.571308 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.571000    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-host-var-lib-cni-bin\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.572219 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.571025    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-host-run-multus-certs\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.572219 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.571051    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22bb6\" (UniqueName: \"kubernetes.io/projected/a140ffcb-b59c-4e55-95e8-e9ed589396b1-kube-api-access-22bb6\") pod \"network-metrics-daemon-m8bvz\" (UID: \"a140ffcb-b59c-4e55-95e8-e9ed589396b1\") " pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:03.572219 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.571059    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 15:42:03.572219 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.571102    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 15:42:03.572219 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.571159    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 15:42:03.572219 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.571162    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 15:42:03.572219 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.571835    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 15:42:03.572219 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.572104    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 15:42:03.572219 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.572105    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mr4ll\""
Sep 10 15:42:03.596015 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.595986    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 15:42:03.634415 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.634375    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-87hps"
Sep 10 15:42:03.645409 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.645374    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-87hps"
Sep 10 15:42:03.660373 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.660344    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 15:42:03.671968 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.671929    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-host\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.671968 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.671974    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-multus-conf-dir\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.672249 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672000    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1601814-5e66-4f97-8165-920af19fb9a4-multus-daemon-config\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.672249 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672032    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-host-slash\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.672249 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672056    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-host-run-netns\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.672249 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672059    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-host\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.672249 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672080    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-multus-conf-dir\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.672249 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672080    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4e0052e2-036e-47d2-9881-5d69873ddff4-etc-selinux\") pod \"aws-ebs-csi-driver-node-sglsh\" (UID: \"4e0052e2-036e-47d2-9881-5d69873ddff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.672249 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672125    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skmlt\" (UniqueName: \"kubernetes.io/projected/9cde9140-e403-43b0-bbda-d78cd59ce6d1-kube-api-access-skmlt\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.672249 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672142    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6f2f68d-170c-4567-ba06-b05bea2cb642-host\") pod \"node-ca-k6fhq\" (UID: \"e6f2f68d-170c-4567-ba06-b05bea2cb642\") " pod="openshift-image-registry/node-ca-k6fhq"
Sep 10 15:42:03.672249 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672241    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e30c4619-7a14-46c2-b2e8-f18c219cfa66-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bjjkl\" (UID: \"e30c4619-7a14-46c2-b2e8-f18c219cfa66\") " pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.672741 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672262    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6f2f68d-170c-4567-ba06-b05bea2cb642-host\") pod \"node-ca-k6fhq\" (UID: \"e6f2f68d-170c-4567-ba06-b05bea2cb642\") " pod="openshift-image-registry/node-ca-k6fhq"
Sep 10 15:42:03.672741 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672275    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d465t\" (UniqueName: \"kubernetes.io/projected/e30c4619-7a14-46c2-b2e8-f18c219cfa66-kube-api-access-d465t\") pod \"multus-additional-cni-plugins-bjjkl\" (UID: \"e30c4619-7a14-46c2-b2e8-f18c219cfa66\") " pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.672741 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672307    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xbhk\" (UniqueName: \"kubernetes.io/projected/24639f5d-eda2-4bf4-a4a4-112ec08b366a-kube-api-access-4xbhk\") pod \"iptables-alerter-k5bnn\" (UID: \"24639f5d-eda2-4bf4-a4a4-112ec08b366a\") " pod="openshift-network-operator/iptables-alerter-k5bnn"
Sep 10 15:42:03.672741 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672331    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-etc-sysctl-d\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.672741 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672351    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-lib-modules\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.672741 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672376    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e30c4619-7a14-46c2-b2e8-f18c219cfa66-cnibin\") pod \"multus-additional-cni-plugins-bjjkl\" (UID: \"e30c4619-7a14-46c2-b2e8-f18c219cfa66\") " pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.672741 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672382    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4e0052e2-036e-47d2-9881-5d69873ddff4-etc-selinux\") pod \"aws-ebs-csi-driver-node-sglsh\" (UID: \"4e0052e2-036e-47d2-9881-5d69873ddff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.672741 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672401    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-system-cni-dir\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.672741 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672426    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-host-run-multus-certs\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.672741 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672451    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-var-lib-openvswitch\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.672741 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672497    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-ovn-node-metrics-cert\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.672741 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672524    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e30c4619-7a14-46c2-b2e8-f18c219cfa66-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bjjkl\" (UID: \"e30c4619-7a14-46c2-b2e8-f18c219cfa66\") " pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.672741 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672556    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22bb6\" (UniqueName: \"kubernetes.io/projected/a140ffcb-b59c-4e55-95e8-e9ed589396b1-kube-api-access-22bb6\") pod \"network-metrics-daemon-m8bvz\" (UID: \"a140ffcb-b59c-4e55-95e8-e9ed589396b1\") " pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:03.672741 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672582    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-host-kubelet\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.672741 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672607    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-etc-openvswitch\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.672741 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672648    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-run-openvswitch\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.672741 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672669    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-run-ovn\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.673644 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672686    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-node-log\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.673644 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672703    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e6f2f68d-170c-4567-ba06-b05bea2cb642-serviceca\") pod \"node-ca-k6fhq\" (UID: \"e6f2f68d-170c-4567-ba06-b05bea2cb642\") " pod="openshift-image-registry/node-ca-k6fhq"
Sep 10 15:42:03.673644 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672723    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-host-var-lib-cni-multus\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.673644 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672765    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/24639f5d-eda2-4bf4-a4a4-112ec08b366a-iptables-alerter-script\") pod \"iptables-alerter-k5bnn\" (UID: \"24639f5d-eda2-4bf4-a4a4-112ec08b366a\") " pod="openshift-network-operator/iptables-alerter-k5bnn"
Sep 10 15:42:03.673644 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672816    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-env-overrides\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.673644 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672840    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1601814-5e66-4f97-8165-920af19fb9a4-cni-binary-copy\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.673644 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672864    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-system-cni-dir\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.673644 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672871    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-run-systemd\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.673644 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672908    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-host-run-ovn-kubernetes\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.673644 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672942    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-var-lib-kubelet\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.673644 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672967    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9cde9140-e403-43b0-bbda-d78cd59ce6d1-etc-tuned\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.673644 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.672994    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e30c4619-7a14-46c2-b2e8-f18c219cfa66-cni-binary-copy\") pod \"multus-additional-cni-plugins-bjjkl\" (UID: \"e30c4619-7a14-46c2-b2e8-f18c219cfa66\") " pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.673644 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.673020    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs\") pod \"network-metrics-daemon-m8bvz\" (UID: \"a140ffcb-b59c-4e55-95e8-e9ed589396b1\") " pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:03.673644 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.673046    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-systemd-units\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.673644 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.673073    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgz7t\" (UniqueName: \"kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t\") pod \"network-check-target-g5l6v\" (UID: \"f5ced900-9635-4cf4-9390-7396c3ef86a7\") " pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:03.673644 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.673108    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzxwl\" (UniqueName: \"kubernetes.io/projected/4e0052e2-036e-47d2-9881-5d69873ddff4-kube-api-access-qzxwl\") pod \"aws-ebs-csi-driver-node-sglsh\" (UID: \"4e0052e2-036e-47d2-9881-5d69873ddff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.673644 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.673135    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e30c4619-7a14-46c2-b2e8-f18c219cfa66-system-cni-dir\") pod \"multus-additional-cni-plugins-bjjkl\" (UID: \"e30c4619-7a14-46c2-b2e8-f18c219cfa66\") " pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.674427 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.673161    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-host-var-lib-kubelet\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.674427 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.673186    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-hostroot\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.674427 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.673236    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4e0052e2-036e-47d2-9881-5d69873ddff4-socket-dir\") pod \"aws-ebs-csi-driver-node-sglsh\" (UID: \"4e0052e2-036e-47d2-9881-5d69873ddff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.674427 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.673254    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1601814-5e66-4f97-8165-920af19fb9a4-multus-daemon-config\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.674427 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.673374    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-run\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.674427 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.673386    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-etc-sysctl-d\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.674427 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.673433    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e6f2f68d-170c-4567-ba06-b05bea2cb642-serviceca\") pod \"node-ca-k6fhq\" (UID: \"e6f2f68d-170c-4567-ba06-b05bea2cb642\") " pod="openshift-image-registry/node-ca-k6fhq"
Sep 10 15:42:03.674427 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.673439    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-host-var-lib-cni-multus\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.674427 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.673491    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-host-run-multus-certs\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.674427 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.673563    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-lib-modules\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.674427 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.673262    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-run\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.674427 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.673996    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e30c4619-7a14-46c2-b2e8-f18c219cfa66-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bjjkl\" (UID: \"e30c4619-7a14-46c2-b2e8-f18c219cfa66\") " pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.675006 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.674433    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1601814-5e66-4f97-8165-920af19fb9a4-cni-binary-copy\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.675006 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.674752    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e30c4619-7a14-46c2-b2e8-f18c219cfa66-system-cni-dir\") pod \"multus-additional-cni-plugins-bjjkl\" (UID: \"e30c4619-7a14-46c2-b2e8-f18c219cfa66\") " pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.675006 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.674769    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 15:42:03.675006 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.674805    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-hostroot\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.675006 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.674881    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cdtx\" (UniqueName: \"kubernetes.io/projected/e6f2f68d-170c-4567-ba06-b05bea2cb642-kube-api-access-2cdtx\") pod \"node-ca-k6fhq\" (UID: \"e6f2f68d-170c-4567-ba06-b05bea2cb642\") " pod="openshift-image-registry/node-ca-k6fhq"
Sep 10 15:42:03.675006 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.674891    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4e0052e2-036e-47d2-9881-5d69873ddff4-socket-dir\") pod \"aws-ebs-csi-driver-node-sglsh\" (UID: \"4e0052e2-036e-47d2-9881-5d69873ddff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.675006 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.674919    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-log-socket\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.675006 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.674923    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e30c4619-7a14-46c2-b2e8-f18c219cfa66-cni-binary-copy\") pod \"multus-additional-cni-plugins-bjjkl\" (UID: \"e30c4619-7a14-46c2-b2e8-f18c219cfa66\") " pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.675006 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.674954    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-etc-kubernetes\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.675006 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.674970    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-host-var-lib-kubelet\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.675006 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.674990    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-host-cni-bin\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.675006 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.675007    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e30c4619-7a14-46c2-b2e8-f18c219cfa66-cnibin\") pod \"multus-additional-cni-plugins-bjjkl\" (UID: \"e30c4619-7a14-46c2-b2e8-f18c219cfa66\") " pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.675646 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.675026    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0cad1bb8-0f30-4ca9-ad4a-fbf4f70122fb-konnectivity-ca\") pod \"konnectivity-agent-q7c8n\" (UID: \"0cad1bb8-0f30-4ca9-ad4a-fbf4f70122fb\") " pod="kube-system/konnectivity-agent-q7c8n"
Sep 10 15:42:03.675646 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.675150    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9cde9140-e403-43b0-bbda-d78cd59ce6d1-tmp\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.675646 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.675169    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4e0052e2-036e-47d2-9881-5d69873ddff4-device-dir\") pod \"aws-ebs-csi-driver-node-sglsh\" (UID: \"4e0052e2-036e-47d2-9881-5d69873ddff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.675646 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.675201    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-etc-kubernetes\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.675646 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:03.675305    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:03.675646 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.675449    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-host-run-netns\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.675646 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.675488    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.675646 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.675557    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4e0052e2-036e-47d2-9881-5d69873ddff4-device-dir\") pod \"aws-ebs-csi-driver-node-sglsh\" (UID: \"4e0052e2-036e-47d2-9881-5d69873ddff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.675646 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.675604    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-ovnkube-script-lib\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.676114 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:03.675661    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs podName:a140ffcb-b59c-4e55-95e8-e9ed589396b1 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:04.175611702 +0000 UTC m=+2.979930962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs") pod "network-metrics-daemon-m8bvz" (UID: "a140ffcb-b59c-4e55-95e8-e9ed589396b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:03.676114 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.675818    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-host-run-netns\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.676114 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.675834    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-var-lib-kubelet\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.676114 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.675862    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8f29f7a6-d451-487d-92f1-84c47bf32d5c-kubelet-config\") pod \"global-pull-secret-syncer-znbph\" (UID: \"8f29f7a6-d451-487d-92f1-84c47bf32d5c\") " pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:03.676114 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.675994    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-etc-kubernetes\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.676114 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.676033    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-host-var-lib-cni-bin\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.676114 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.676091    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-etc-kubernetes\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.676114 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.676106    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e30c4619-7a14-46c2-b2e8-f18c219cfa66-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bjjkl\" (UID: \"e30c4619-7a14-46c2-b2e8-f18c219cfa66\") " pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.676452 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.676157    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4e0052e2-036e-47d2-9881-5d69873ddff4-sys-fs\") pod \"aws-ebs-csi-driver-node-sglsh\" (UID: \"4e0052e2-036e-47d2-9881-5d69873ddff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.676452 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.676198    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-host-var-lib-cni-bin\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.676452 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.676252    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-etc-modprobe-d\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.676452 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.676286    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-multus-cni-dir\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.676452 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.676337    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-host-run-k8s-cni-cncf-io\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.676452 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.676365    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/24639f5d-eda2-4bf4-a4a4-112ec08b366a-host-slash\") pod \"iptables-alerter-k5bnn\" (UID: \"24639f5d-eda2-4bf4-a4a4-112ec08b366a\") " pod="openshift-network-operator/iptables-alerter-k5bnn"
Sep 10 15:42:03.676452 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.676395    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-host-cni-netd\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.676452 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.676428    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0cad1bb8-0f30-4ca9-ad4a-fbf4f70122fb-agent-certs\") pod \"konnectivity-agent-q7c8n\" (UID: \"0cad1bb8-0f30-4ca9-ad4a-fbf4f70122fb\") " pod="kube-system/konnectivity-agent-q7c8n"
Sep 10 15:42:03.676734 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.676483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret\") pod \"global-pull-secret-syncer-znbph\" (UID: \"8f29f7a6-d451-487d-92f1-84c47bf32d5c\") " pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:03.676734 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.676518    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4e0052e2-036e-47d2-9881-5d69873ddff4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sglsh\" (UID: \"4e0052e2-036e-47d2-9881-5d69873ddff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.676734 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.676543    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4e0052e2-036e-47d2-9881-5d69873ddff4-registration-dir\") pod \"aws-ebs-csi-driver-node-sglsh\" (UID: \"4e0052e2-036e-47d2-9881-5d69873ddff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.676734 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.676573    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-etc-systemd\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.676734 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.676609    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-sys\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.676734 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.676638    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-multus-socket-dir-parent\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.676734 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.676668    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmvbb\" (UniqueName: \"kubernetes.io/projected/e1601814-5e66-4f97-8165-920af19fb9a4-kube-api-access-fmvbb\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.676734 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.676698    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-ovnkube-config\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.677076 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.677056    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dj7g\" (UniqueName: \"kubernetes.io/projected/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-kube-api-access-4dj7g\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.677123 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.677103    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-etc-sysconfig\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.677156 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.677134    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e30c4619-7a14-46c2-b2e8-f18c219cfa66-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bjjkl\" (UID: \"e30c4619-7a14-46c2-b2e8-f18c219cfa66\") " pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.677190 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.677166    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8f29f7a6-d451-487d-92f1-84c47bf32d5c-dbus\") pod \"global-pull-secret-syncer-znbph\" (UID: \"8f29f7a6-d451-487d-92f1-84c47bf32d5c\") " pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:03.677288 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.677199    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-etc-sysctl-conf\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.677288 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.677257    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-cnibin\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.677355 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.677303    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4e0052e2-036e-47d2-9881-5d69873ddff4-registration-dir\") pod \"aws-ebs-csi-driver-node-sglsh\" (UID: \"4e0052e2-036e-47d2-9881-5d69873ddff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.677389 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.677380    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-cnibin\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.677460 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.677446    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-multus-cni-dir\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.677518 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.677502    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4e0052e2-036e-47d2-9881-5d69873ddff4-sys-fs\") pod \"aws-ebs-csi-driver-node-sglsh\" (UID: \"4e0052e2-036e-47d2-9881-5d69873ddff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.677758 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.677567    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-host-run-k8s-cni-cncf-io\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.677758 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.677653    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4e0052e2-036e-47d2-9881-5d69873ddff4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sglsh\" (UID: \"4e0052e2-036e-47d2-9881-5d69873ddff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.677870 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.677704    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-etc-sysconfig\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.677870 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.677783    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e30c4619-7a14-46c2-b2e8-f18c219cfa66-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bjjkl\" (UID: \"e30c4619-7a14-46c2-b2e8-f18c219cfa66\") " pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.677971 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.677945    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-etc-systemd\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.677971 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.677958    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-etc-sysctl-conf\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.678071 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.678018    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-sys\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.678169 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.678146    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1601814-5e66-4f97-8165-920af19fb9a4-multus-socket-dir-parent\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.679501 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.678607    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9cde9140-e403-43b0-bbda-d78cd59ce6d1-etc-modprobe-d\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.679870 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.679848    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9cde9140-e403-43b0-bbda-d78cd59ce6d1-etc-tuned\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.681582 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.681555    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9cde9140-e403-43b0-bbda-d78cd59ce6d1-tmp\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.694178 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.694127    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skmlt\" (UniqueName: \"kubernetes.io/projected/9cde9140-e403-43b0-bbda-d78cd59ce6d1-kube-api-access-skmlt\") pod \"tuned-66dx8\" (UID: \"9cde9140-e403-43b0-bbda-d78cd59ce6d1\") " pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.697490 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.697455    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzxwl\" (UniqueName: \"kubernetes.io/projected/4e0052e2-036e-47d2-9881-5d69873ddff4-kube-api-access-qzxwl\") pod \"aws-ebs-csi-driver-node-sglsh\" (UID: \"4e0052e2-036e-47d2-9881-5d69873ddff4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.703098 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.703069    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d465t\" (UniqueName: \"kubernetes.io/projected/e30c4619-7a14-46c2-b2e8-f18c219cfa66-kube-api-access-d465t\") pod \"multus-additional-cni-plugins-bjjkl\" (UID: \"e30c4619-7a14-46c2-b2e8-f18c219cfa66\") " pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.703932 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.703913    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22bb6\" (UniqueName: \"kubernetes.io/projected/a140ffcb-b59c-4e55-95e8-e9ed589396b1-kube-api-access-22bb6\") pod \"network-metrics-daemon-m8bvz\" (UID: \"a140ffcb-b59c-4e55-95e8-e9ed589396b1\") " pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:03.704731 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.704708    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmvbb\" (UniqueName: \"kubernetes.io/projected/e1601814-5e66-4f97-8165-920af19fb9a4-kube-api-access-fmvbb\") pod \"multus-pl75t\" (UID: \"e1601814-5e66-4f97-8165-920af19fb9a4\") " pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.704828 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.704750    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cdtx\" (UniqueName: \"kubernetes.io/projected/e6f2f68d-170c-4567-ba06-b05bea2cb642-kube-api-access-2cdtx\") pod \"node-ca-k6fhq\" (UID: \"e6f2f68d-170c-4567-ba06-b05bea2cb642\") " pod="openshift-image-registry/node-ca-k6fhq"
Sep 10 15:42:03.778330 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778231    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-log-socket\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.778330 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778277    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-host-cni-bin\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.778330 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778304    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0cad1bb8-0f30-4ca9-ad4a-fbf4f70122fb-konnectivity-ca\") pod \"konnectivity-agent-q7c8n\" (UID: \"0cad1bb8-0f30-4ca9-ad4a-fbf4f70122fb\") " pod="kube-system/konnectivity-agent-q7c8n"
Sep 10 15:42:03.778634 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778332    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-log-socket\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.778634 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778338    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.778634 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778381    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.778634 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778384    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-host-cni-bin\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.778634 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778398    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-ovnkube-script-lib\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.778634 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778453    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8f29f7a6-d451-487d-92f1-84c47bf32d5c-kubelet-config\") pod \"global-pull-secret-syncer-znbph\" (UID: \"8f29f7a6-d451-487d-92f1-84c47bf32d5c\") " pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:03.778634 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778494    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/24639f5d-eda2-4bf4-a4a4-112ec08b366a-host-slash\") pod \"iptables-alerter-k5bnn\" (UID: \"24639f5d-eda2-4bf4-a4a4-112ec08b366a\") " pod="openshift-network-operator/iptables-alerter-k5bnn"
Sep 10 15:42:03.778634 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778519    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-host-cni-netd\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.778634 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778547    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0cad1bb8-0f30-4ca9-ad4a-fbf4f70122fb-agent-certs\") pod \"konnectivity-agent-q7c8n\" (UID: \"0cad1bb8-0f30-4ca9-ad4a-fbf4f70122fb\") " pod="kube-system/konnectivity-agent-q7c8n"
Sep 10 15:42:03.778634 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778541    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8f29f7a6-d451-487d-92f1-84c47bf32d5c-kubelet-config\") pod \"global-pull-secret-syncer-znbph\" (UID: \"8f29f7a6-d451-487d-92f1-84c47bf32d5c\") " pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:03.778634 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778573    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret\") pod \"global-pull-secret-syncer-znbph\" (UID: \"8f29f7a6-d451-487d-92f1-84c47bf32d5c\") " pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:03.778634 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778606    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-ovnkube-config\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.778634 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778637    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-host-cni-netd\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779333 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778639    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dj7g\" (UniqueName: \"kubernetes.io/projected/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-kube-api-access-4dj7g\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779333 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778679    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8f29f7a6-d451-487d-92f1-84c47bf32d5c-dbus\") pod \"global-pull-secret-syncer-znbph\" (UID: \"8f29f7a6-d451-487d-92f1-84c47bf32d5c\") " pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:03.779333 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778702    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-host-slash\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779333 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778718    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-host-run-netns\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779333 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778737    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xbhk\" (UniqueName: \"kubernetes.io/projected/24639f5d-eda2-4bf4-a4a4-112ec08b366a-kube-api-access-4xbhk\") pod \"iptables-alerter-k5bnn\" (UID: \"24639f5d-eda2-4bf4-a4a4-112ec08b366a\") " pod="openshift-network-operator/iptables-alerter-k5bnn"
Sep 10 15:42:03.779333 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778765    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-var-lib-openvswitch\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779333 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778783    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-ovn-node-metrics-cert\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779333 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778800    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-host-kubelet\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779333 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778814    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-etc-openvswitch\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779333 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778828    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-run-openvswitch\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779333 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778848    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-run-ovn\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779333 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778881    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-node-log\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779333 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778898    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/24639f5d-eda2-4bf4-a4a4-112ec08b366a-iptables-alerter-script\") pod \"iptables-alerter-k5bnn\" (UID: \"24639f5d-eda2-4bf4-a4a4-112ec08b366a\") " pod="openshift-network-operator/iptables-alerter-k5bnn"
Sep 10 15:42:03.779333 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.778996    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0cad1bb8-0f30-4ca9-ad4a-fbf4f70122fb-konnectivity-ca\") pod \"konnectivity-agent-q7c8n\" (UID: \"0cad1bb8-0f30-4ca9-ad4a-fbf4f70122fb\") " pod="kube-system/konnectivity-agent-q7c8n"
Sep 10 15:42:03.779333 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.779021    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-ovnkube-script-lib\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779333 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.779059    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-var-lib-openvswitch\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779333 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.779102    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/24639f5d-eda2-4bf4-a4a4-112ec08b366a-host-slash\") pod \"iptables-alerter-k5bnn\" (UID: \"24639f5d-eda2-4bf4-a4a4-112ec08b366a\") " pod="openshift-network-operator/iptables-alerter-k5bnn"
Sep 10 15:42:03.779333 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:03.779118    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:03.779955 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.779156    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8f29f7a6-d451-487d-92f1-84c47bf32d5c-dbus\") pod \"global-pull-secret-syncer-znbph\" (UID: \"8f29f7a6-d451-487d-92f1-84c47bf32d5c\") " pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:03.779955 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:03.779177    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret podName:8f29f7a6-d451-487d-92f1-84c47bf32d5c nodeName:}" failed. No retries permitted until 2026-09-10 15:42:04.279157612 +0000 UTC m=+3.083476869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret") pod "global-pull-secret-syncer-znbph" (UID: "8f29f7a6-d451-487d-92f1-84c47bf32d5c") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:03.779955 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.779197    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-host-slash\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779955 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.779257    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-host-run-netns\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779955 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.779367    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/24639f5d-eda2-4bf4-a4a4-112ec08b366a-iptables-alerter-script\") pod \"iptables-alerter-k5bnn\" (UID: \"24639f5d-eda2-4bf4-a4a4-112ec08b366a\") " pod="openshift-network-operator/iptables-alerter-k5bnn"
Sep 10 15:42:03.779955 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.779478    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-run-openvswitch\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779955 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.779522    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-host-kubelet\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779955 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.779563    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-etc-openvswitch\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779955 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.779603    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-run-ovn\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779955 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.779641    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-node-log\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779955 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.779673    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-env-overrides\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779955 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.779703    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-run-systemd\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779955 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.779729    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-host-run-ovn-kubernetes\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779955 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.779774    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-systemd-units\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779955 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.779802    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgz7t\" (UniqueName: \"kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t\") pod \"network-check-target-g5l6v\" (UID: \"f5ced900-9635-4cf4-9390-7396c3ef86a7\") " pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:03.779955 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.779830    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-ovnkube-config\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.779955 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.779884    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-systemd-units\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.780667 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.779885    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-run-systemd\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.780667 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.779918    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-host-run-ovn-kubernetes\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.780667 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.780216    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-env-overrides\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.781838 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.781812    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0cad1bb8-0f30-4ca9-ad4a-fbf4f70122fb-agent-certs\") pod \"konnectivity-agent-q7c8n\" (UID: \"0cad1bb8-0f30-4ca9-ad4a-fbf4f70122fb\") " pod="kube-system/konnectivity-agent-q7c8n"
Sep 10 15:42:03.782377 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.782357    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-ovn-node-metrics-cert\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.787559 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:03.787538    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:42:03.787660 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:03.787564    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:42:03.787660 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:03.787579    2564 projected.go:196] Error preparing data for projected volume kube-api-access-dgz7t for pod openshift-network-diagnostics/network-check-target-g5l6v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:03.787739 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:03.787679    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t podName:f5ced900-9635-4cf4-9390-7396c3ef86a7 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:04.287652181 +0000 UTC m=+3.091971427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dgz7t" (UniqueName: "kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t") pod "network-check-target-g5l6v" (UID: "f5ced900-9635-4cf4-9390-7396c3ef86a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:03.788860 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.788840    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dj7g\" (UniqueName: \"kubernetes.io/projected/2f3e30ed-505d-4f2b-a1a7-3033e1b1d718-kube-api-access-4dj7g\") pod \"ovnkube-node-4xzlm\" (UID: \"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718\") " pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.789119 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.789099    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xbhk\" (UniqueName: \"kubernetes.io/projected/24639f5d-eda2-4bf4-a4a4-112ec08b366a-kube-api-access-4xbhk\") pod \"iptables-alerter-k5bnn\" (UID: \"24639f5d-eda2-4bf4-a4a4-112ec08b366a\") " pod="openshift-network-operator/iptables-alerter-k5bnn"
Sep 10 15:42:03.867055 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.866784    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k6fhq"
Sep 10 15:42:03.876202 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.876178    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh"
Sep 10 15:42:03.884036 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.884017    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bjjkl"
Sep 10 15:42:03.888902 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.888881    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pl75t"
Sep 10 15:42:03.895652 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.895629    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-66dx8"
Sep 10 15:42:03.903452 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.903429    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k5bnn"
Sep 10 15:42:03.909938 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.909919    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:03.914270 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:03.914249    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q7c8n"
Sep 10 15:42:04.183389 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:04.183339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs\") pod \"network-metrics-daemon-m8bvz\" (UID: \"a140ffcb-b59c-4e55-95e8-e9ed589396b1\") " pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:04.183562 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:04.183503    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:04.183622 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:04.183573    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs podName:a140ffcb-b59c-4e55-95e8-e9ed589396b1 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:05.183556642 +0000 UTC m=+3.987875888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs") pod "network-metrics-daemon-m8bvz" (UID: "a140ffcb-b59c-4e55-95e8-e9ed589396b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:04.237002 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:04.236970    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f2f68d_170c_4567_ba06_b05bea2cb642.slice/crio-06ed0e4ba2c7165fa35a76327d216d2db280b208d23a7b6f16e35f6f7996b1dd WatchSource:0}: Error finding container 06ed0e4ba2c7165fa35a76327d216d2db280b208d23a7b6f16e35f6f7996b1dd: Status 404 returned error can't find the container with id 06ed0e4ba2c7165fa35a76327d216d2db280b208d23a7b6f16e35f6f7996b1dd
Sep 10 15:42:04.237914 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:04.237891    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24639f5d_eda2_4bf4_a4a4_112ec08b366a.slice/crio-481a90c8bb3c7efa811e68b95ff5f14ad51f3b0ea3ae471aa74a6736f3073405 WatchSource:0}: Error finding container 481a90c8bb3c7efa811e68b95ff5f14ad51f3b0ea3ae471aa74a6736f3073405: Status 404 returned error can't find the container with id 481a90c8bb3c7efa811e68b95ff5f14ad51f3b0ea3ae471aa74a6736f3073405
Sep 10 15:42:04.248783 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:04.248758    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3e30ed_505d_4f2b_a1a7_3033e1b1d718.slice/crio-29eddfaa216da014fef3f2e8d161d7ea4343456d282abeda05f0a219e80dd107 WatchSource:0}: Error finding container 29eddfaa216da014fef3f2e8d161d7ea4343456d282abeda05f0a219e80dd107: Status 404 returned error can't find the container with id 29eddfaa216da014fef3f2e8d161d7ea4343456d282abeda05f0a219e80dd107
Sep 10 15:42:04.251028 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:04.251002    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0052e2_036e_47d2_9881_5d69873ddff4.slice/crio-cb566ff0d045bc32fd5dd456adcfb805fb1aaf450dc29c87e09e918d6618ac5d WatchSource:0}: Error finding container cb566ff0d045bc32fd5dd456adcfb805fb1aaf450dc29c87e09e918d6618ac5d: Status 404 returned error can't find the container with id cb566ff0d045bc32fd5dd456adcfb805fb1aaf450dc29c87e09e918d6618ac5d
Sep 10 15:42:04.252407 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:04.252388    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cde9140_e403_43b0_bbda_d78cd59ce6d1.slice/crio-d69fa4a6bb91b1292979135064be6813f0181a359e8da7dff8eb380fc5d50937 WatchSource:0}: Error finding container d69fa4a6bb91b1292979135064be6813f0181a359e8da7dff8eb380fc5d50937: Status 404 returned error can't find the container with id d69fa4a6bb91b1292979135064be6813f0181a359e8da7dff8eb380fc5d50937
Sep 10 15:42:04.253573 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:04.253544    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1601814_5e66_4f97_8165_920af19fb9a4.slice/crio-f3a9615e2db93ee304c1e7755cff581c668e397e61588b8276c84bb585e6cd5b WatchSource:0}: Error finding container f3a9615e2db93ee304c1e7755cff581c668e397e61588b8276c84bb585e6cd5b: Status 404 returned error can't find the container with id f3a9615e2db93ee304c1e7755cff581c668e397e61588b8276c84bb585e6cd5b
Sep 10 15:42:04.254892 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:04.254873    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cad1bb8_0f30_4ca9_ad4a_fbf4f70122fb.slice/crio-881feb4555fce42b6114148eb8cf11ead51a21de53b59074aaf124a53e02245f WatchSource:0}: Error finding container 881feb4555fce42b6114148eb8cf11ead51a21de53b59074aaf124a53e02245f: Status 404 returned error can't find the container with id 881feb4555fce42b6114148eb8cf11ead51a21de53b59074aaf124a53e02245f
Sep 10 15:42:04.255545 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:04.255521    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode30c4619_7a14_46c2_b2e8_f18c219cfa66.slice/crio-9471ac95c4657061270fba615e4cbc8c8aaf11b7ad59fb032d648c377414a2a5 WatchSource:0}: Error finding container 9471ac95c4657061270fba615e4cbc8c8aaf11b7ad59fb032d648c377414a2a5: Status 404 returned error can't find the container with id 9471ac95c4657061270fba615e4cbc8c8aaf11b7ad59fb032d648c377414a2a5
Sep 10 15:42:04.265496 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:04.265472    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dcad2a033892fede0cdda710bab032c.slice/crio-5e4e73048b6d8550afdb4a06e647eaa4c73d4c286bb1dab90a9a9ecc10896197 WatchSource:0}: Error finding container 5e4e73048b6d8550afdb4a06e647eaa4c73d4c286bb1dab90a9a9ecc10896197: Status 404 returned error can't find the container with id 5e4e73048b6d8550afdb4a06e647eaa4c73d4c286bb1dab90a9a9ecc10896197
Sep 10 15:42:04.283874 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:04.283845    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret\") pod \"global-pull-secret-syncer-znbph\" (UID: \"8f29f7a6-d451-487d-92f1-84c47bf32d5c\") " pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:04.284011 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:04.283988    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:04.284086 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:04.284073    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret podName:8f29f7a6-d451-487d-92f1-84c47bf32d5c nodeName:}" failed. No retries permitted until 2026-09-10 15:42:05.284042566 +0000 UTC m=+4.088361826 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret") pod "global-pull-secret-syncer-znbph" (UID: "8f29f7a6-d451-487d-92f1-84c47bf32d5c") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:04.385522 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:04.385480    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgz7t\" (UniqueName: \"kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t\") pod \"network-check-target-g5l6v\" (UID: \"f5ced900-9635-4cf4-9390-7396c3ef86a7\") " pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:04.385643 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:04.385602    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:42:04.385643 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:04.385617    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:42:04.385643 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:04.385628    2564 projected.go:196] Error preparing data for projected volume kube-api-access-dgz7t for pod openshift-network-diagnostics/network-check-target-g5l6v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:04.385772 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:04.385685    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t podName:f5ced900-9635-4cf4-9390-7396c3ef86a7 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:05.385669003 +0000 UTC m=+4.189988243 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dgz7t" (UniqueName: "kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t") pod "network-check-target-g5l6v" (UID: "f5ced900-9635-4cf4-9390-7396c3ef86a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:04.647935 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:04.647837    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:37:04 +0000 UTC" deadline="2028-05-12 07:17:48.190532919 +0000 UTC"
Sep 10 15:42:04.647935 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:04.647878    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14631h35m43.542658733s"
Sep 10 15:42:04.673614 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:04.673577    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-145.ec2.internal" event={"ID":"8dcad2a033892fede0cdda710bab032c","Type":"ContainerStarted","Data":"5e4e73048b6d8550afdb4a06e647eaa4c73d4c286bb1dab90a9a9ecc10896197"}
Sep 10 15:42:04.674959 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:04.674922    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bjjkl" event={"ID":"e30c4619-7a14-46c2-b2e8-f18c219cfa66","Type":"ContainerStarted","Data":"9471ac95c4657061270fba615e4cbc8c8aaf11b7ad59fb032d648c377414a2a5"}
Sep 10 15:42:04.676974 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:04.676937    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q7c8n" event={"ID":"0cad1bb8-0f30-4ca9-ad4a-fbf4f70122fb","Type":"ContainerStarted","Data":"881feb4555fce42b6114148eb8cf11ead51a21de53b59074aaf124a53e02245f"}
Sep 10 15:42:04.678376 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:04.678333    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pl75t" event={"ID":"e1601814-5e66-4f97-8165-920af19fb9a4","Type":"ContainerStarted","Data":"f3a9615e2db93ee304c1e7755cff581c668e397e61588b8276c84bb585e6cd5b"}
Sep 10 15:42:04.680013 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:04.679985    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-66dx8" event={"ID":"9cde9140-e403-43b0-bbda-d78cd59ce6d1","Type":"ContainerStarted","Data":"d69fa4a6bb91b1292979135064be6813f0181a359e8da7dff8eb380fc5d50937"}
Sep 10 15:42:04.681357 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:04.681308    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k5bnn" event={"ID":"24639f5d-eda2-4bf4-a4a4-112ec08b366a","Type":"ContainerStarted","Data":"481a90c8bb3c7efa811e68b95ff5f14ad51f3b0ea3ae471aa74a6736f3073405"}
Sep 10 15:42:04.682961 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:04.682745    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k6fhq" event={"ID":"e6f2f68d-170c-4567-ba06-b05bea2cb642","Type":"ContainerStarted","Data":"06ed0e4ba2c7165fa35a76327d216d2db280b208d23a7b6f16e35f6f7996b1dd"}
Sep 10 15:42:04.689555 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:04.689512    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-145.ec2.internal" event={"ID":"05d404cb45a3a313356b08f70399e53d","Type":"ContainerStarted","Data":"c0bf69bd2051ce263ffa1c038490d550da99f5806742bfc6bedb9220ca61d869"}
Sep 10 15:42:04.692399 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:04.692374    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh" event={"ID":"4e0052e2-036e-47d2-9881-5d69873ddff4","Type":"ContainerStarted","Data":"cb566ff0d045bc32fd5dd456adcfb805fb1aaf450dc29c87e09e918d6618ac5d"}
Sep 10 15:42:04.693829 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:04.693781    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm" event={"ID":"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718","Type":"ContainerStarted","Data":"29eddfaa216da014fef3f2e8d161d7ea4343456d282abeda05f0a219e80dd107"}
Sep 10 15:42:05.192130 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:05.192070    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs\") pod \"network-metrics-daemon-m8bvz\" (UID: \"a140ffcb-b59c-4e55-95e8-e9ed589396b1\") " pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:05.192347 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:05.192305    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:05.192432 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:05.192379    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs podName:a140ffcb-b59c-4e55-95e8-e9ed589396b1 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:07.192357792 +0000 UTC m=+5.996677048 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs") pod "network-metrics-daemon-m8bvz" (UID: "a140ffcb-b59c-4e55-95e8-e9ed589396b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:05.293349 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:05.293302    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret\") pod \"global-pull-secret-syncer-znbph\" (UID: \"8f29f7a6-d451-487d-92f1-84c47bf32d5c\") " pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:05.293555 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:05.293478    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:05.293555 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:05.293548    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret podName:8f29f7a6-d451-487d-92f1-84c47bf32d5c nodeName:}" failed. No retries permitted until 2026-09-10 15:42:07.293527418 +0000 UTC m=+6.097846664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret") pod "global-pull-secret-syncer-znbph" (UID: "8f29f7a6-d451-487d-92f1-84c47bf32d5c") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:05.394646 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:05.394600    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgz7t\" (UniqueName: \"kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t\") pod \"network-check-target-g5l6v\" (UID: \"f5ced900-9635-4cf4-9390-7396c3ef86a7\") " pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:05.394851 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:05.394836    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:42:05.394925 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:05.394860    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:42:05.394925 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:05.394875    2564 projected.go:196] Error preparing data for projected volume kube-api-access-dgz7t for pod openshift-network-diagnostics/network-check-target-g5l6v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:05.395039 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:05.394939    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t podName:f5ced900-9635-4cf4-9390-7396c3ef86a7 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:07.394919329 +0000 UTC m=+6.199238588 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dgz7t" (UniqueName: "kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t") pod "network-check-target-g5l6v" (UID: "f5ced900-9635-4cf4-9390-7396c3ef86a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:05.649171 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:05.649025    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:37:04 +0000 UTC" deadline="2028-03-31 09:54:16.970907264 +0000 UTC"
Sep 10 15:42:05.649171 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:05.649088    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13626h12m11.321823517s"
Sep 10 15:42:05.674733 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:05.673761    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:05.674733 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:05.673891    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5l6v" podUID="f5ced900-9635-4cf4-9390-7396c3ef86a7"
Sep 10 15:42:05.674733 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:05.674354    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:05.674733 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:05.674458    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-znbph" podUID="8f29f7a6-d451-487d-92f1-84c47bf32d5c"
Sep 10 15:42:05.674733 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:05.674527    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:05.674733 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:05.674612    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8bvz" podUID="a140ffcb-b59c-4e55-95e8-e9ed589396b1"
Sep 10 15:42:07.214386 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:07.214345    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs\") pod \"network-metrics-daemon-m8bvz\" (UID: \"a140ffcb-b59c-4e55-95e8-e9ed589396b1\") " pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:07.214894 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:07.214485    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:07.214894 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:07.214579    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs podName:a140ffcb-b59c-4e55-95e8-e9ed589396b1 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:11.214555306 +0000 UTC m=+10.018874570 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs") pod "network-metrics-daemon-m8bvz" (UID: "a140ffcb-b59c-4e55-95e8-e9ed589396b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:07.315024 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:07.314985    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret\") pod \"global-pull-secret-syncer-znbph\" (UID: \"8f29f7a6-d451-487d-92f1-84c47bf32d5c\") " pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:07.315248 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:07.315162    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:07.315332 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:07.315248    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret podName:8f29f7a6-d451-487d-92f1-84c47bf32d5c nodeName:}" failed. No retries permitted until 2026-09-10 15:42:11.315228677 +0000 UTC m=+10.119547924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret") pod "global-pull-secret-syncer-znbph" (UID: "8f29f7a6-d451-487d-92f1-84c47bf32d5c") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:07.416119 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:07.415435    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgz7t\" (UniqueName: \"kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t\") pod \"network-check-target-g5l6v\" (UID: \"f5ced900-9635-4cf4-9390-7396c3ef86a7\") " pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:07.416119 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:07.415638    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:42:07.416119 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:07.415658    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:42:07.416119 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:07.415672    2564 projected.go:196] Error preparing data for projected volume kube-api-access-dgz7t for pod openshift-network-diagnostics/network-check-target-g5l6v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:07.416119 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:07.415732    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t podName:f5ced900-9635-4cf4-9390-7396c3ef86a7 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:11.415714191 +0000 UTC m=+10.220033438 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dgz7t" (UniqueName: "kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t") pod "network-check-target-g5l6v" (UID: "f5ced900-9635-4cf4-9390-7396c3ef86a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:07.666332 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:07.666240    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:07.666511 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:07.666384    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8bvz" podUID="a140ffcb-b59c-4e55-95e8-e9ed589396b1"
Sep 10 15:42:07.666764 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:07.666741    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:07.666871 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:07.666851    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-znbph" podUID="8f29f7a6-d451-487d-92f1-84c47bf32d5c"
Sep 10 15:42:07.666940 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:07.666904    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:07.667013 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:07.666973    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5l6v" podUID="f5ced900-9635-4cf4-9390-7396c3ef86a7"
Sep 10 15:42:09.666617 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:09.666508    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:09.666617 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:09.666550    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:09.666617 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:09.666508    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:09.667153 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:09.666656    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8bvz" podUID="a140ffcb-b59c-4e55-95e8-e9ed589396b1"
Sep 10 15:42:09.667153 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:09.666750    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5l6v" podUID="f5ced900-9635-4cf4-9390-7396c3ef86a7"
Sep 10 15:42:09.667153 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:09.666821    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-znbph" podUID="8f29f7a6-d451-487d-92f1-84c47bf32d5c"
Sep 10 15:42:11.249176 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:11.249122    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs\") pod \"network-metrics-daemon-m8bvz\" (UID: \"a140ffcb-b59c-4e55-95e8-e9ed589396b1\") " pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:11.249652 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:11.249332    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:11.249652 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:11.249399    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs podName:a140ffcb-b59c-4e55-95e8-e9ed589396b1 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:19.24938094 +0000 UTC m=+18.053700184 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs") pod "network-metrics-daemon-m8bvz" (UID: "a140ffcb-b59c-4e55-95e8-e9ed589396b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:11.350257 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:11.350112    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret\") pod \"global-pull-secret-syncer-znbph\" (UID: \"8f29f7a6-d451-487d-92f1-84c47bf32d5c\") " pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:11.350409 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:11.350279    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:11.350409 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:11.350340    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret podName:8f29f7a6-d451-487d-92f1-84c47bf32d5c nodeName:}" failed. No retries permitted until 2026-09-10 15:42:19.350325604 +0000 UTC m=+18.154644843 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret") pod "global-pull-secret-syncer-znbph" (UID: "8f29f7a6-d451-487d-92f1-84c47bf32d5c") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:11.452063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:11.451444    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgz7t\" (UniqueName: \"kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t\") pod \"network-check-target-g5l6v\" (UID: \"f5ced900-9635-4cf4-9390-7396c3ef86a7\") " pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:11.452063 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:11.451634    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:42:11.452063 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:11.451654    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:42:11.452063 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:11.451666    2564 projected.go:196] Error preparing data for projected volume kube-api-access-dgz7t for pod openshift-network-diagnostics/network-check-target-g5l6v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:11.452063 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:11.451723    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t podName:f5ced900-9635-4cf4-9390-7396c3ef86a7 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:19.451703187 +0000 UTC m=+18.256022449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dgz7t" (UniqueName: "kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t") pod "network-check-target-g5l6v" (UID: "f5ced900-9635-4cf4-9390-7396c3ef86a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:11.667710 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:11.667681    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:11.667880 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:11.667805    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5l6v" podUID="f5ced900-9635-4cf4-9390-7396c3ef86a7"
Sep 10 15:42:11.668196 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:11.668177    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:11.668306 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:11.668286    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-znbph" podUID="8f29f7a6-d451-487d-92f1-84c47bf32d5c"
Sep 10 15:42:11.668372 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:11.668338    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:11.668440 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:11.668421    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8bvz" podUID="a140ffcb-b59c-4e55-95e8-e9ed589396b1"
Sep 10 15:42:13.669357 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:13.669325    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:13.669976 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:13.669444    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-znbph" podUID="8f29f7a6-d451-487d-92f1-84c47bf32d5c"
Sep 10 15:42:13.669976 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:13.669336    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:13.669976 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:13.669537    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5l6v" podUID="f5ced900-9635-4cf4-9390-7396c3ef86a7"
Sep 10 15:42:13.669976 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:13.669325    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:13.669976 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:13.669631    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8bvz" podUID="a140ffcb-b59c-4e55-95e8-e9ed589396b1"
Sep 10 15:42:15.666660 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:15.666563    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:15.666660 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:15.666615    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:15.667195 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:15.666562    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:15.667195 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:15.666708    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5l6v" podUID="f5ced900-9635-4cf4-9390-7396c3ef86a7"
Sep 10 15:42:15.667195 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:15.666815    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-znbph" podUID="8f29f7a6-d451-487d-92f1-84c47bf32d5c"
Sep 10 15:42:15.667195 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:15.666948    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8bvz" podUID="a140ffcb-b59c-4e55-95e8-e9ed589396b1"
Sep 10 15:42:17.666043 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:17.666007    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:17.666507 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:17.666079    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:17.666507 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:17.666128    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:17.666507 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:17.666263    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5l6v" podUID="f5ced900-9635-4cf4-9390-7396c3ef86a7"
Sep 10 15:42:17.666507 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:17.666416    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-znbph" podUID="8f29f7a6-d451-487d-92f1-84c47bf32d5c"
Sep 10 15:42:17.666670 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:17.666551    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8bvz" podUID="a140ffcb-b59c-4e55-95e8-e9ed589396b1"
Sep 10 15:42:19.312559 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:19.312514    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs\") pod \"network-metrics-daemon-m8bvz\" (UID: \"a140ffcb-b59c-4e55-95e8-e9ed589396b1\") " pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:19.313074 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:19.312648    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:19.313074 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:19.312715    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs podName:a140ffcb-b59c-4e55-95e8-e9ed589396b1 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:35.312696162 +0000 UTC m=+34.117015401 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs") pod "network-metrics-daemon-m8bvz" (UID: "a140ffcb-b59c-4e55-95e8-e9ed589396b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:19.412863 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:19.412825    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret\") pod \"global-pull-secret-syncer-znbph\" (UID: \"8f29f7a6-d451-487d-92f1-84c47bf32d5c\") " pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:19.413063 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:19.412991    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:19.413132 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:19.413073    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret podName:8f29f7a6-d451-487d-92f1-84c47bf32d5c nodeName:}" failed. No retries permitted until 2026-09-10 15:42:35.413051724 +0000 UTC m=+34.217370976 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret") pod "global-pull-secret-syncer-znbph" (UID: "8f29f7a6-d451-487d-92f1-84c47bf32d5c") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:19.513968 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:19.513930    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgz7t\" (UniqueName: \"kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t\") pod \"network-check-target-g5l6v\" (UID: \"f5ced900-9635-4cf4-9390-7396c3ef86a7\") " pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:19.514145 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:19.514105    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:42:19.514145 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:19.514125    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:42:19.514145 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:19.514137    2564 projected.go:196] Error preparing data for projected volume kube-api-access-dgz7t for pod openshift-network-diagnostics/network-check-target-g5l6v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:19.514310 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:19.514208    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t podName:f5ced900-9635-4cf4-9390-7396c3ef86a7 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:35.51419492 +0000 UTC m=+34.318514159 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dgz7t" (UniqueName: "kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t") pod "network-check-target-g5l6v" (UID: "f5ced900-9635-4cf4-9390-7396c3ef86a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:19.666659 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:19.666572    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:19.666819 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:19.666572    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:19.666819 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:19.666712    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-znbph" podUID="8f29f7a6-d451-487d-92f1-84c47bf32d5c"
Sep 10 15:42:19.666819 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:19.666585    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:19.666819 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:19.666775    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5l6v" podUID="f5ced900-9635-4cf4-9390-7396c3ef86a7"
Sep 10 15:42:19.666984 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:19.666852    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8bvz" podUID="a140ffcb-b59c-4e55-95e8-e9ed589396b1"
Sep 10 15:42:21.667581 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:21.667539    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:21.668044 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:21.667649    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:21.668044 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:21.667658    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5l6v" podUID="f5ced900-9635-4cf4-9390-7396c3ef86a7"
Sep 10 15:42:21.668044 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:21.667769    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-znbph" podUID="8f29f7a6-d451-487d-92f1-84c47bf32d5c"
Sep 10 15:42:21.668044 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:21.667830    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:21.668044 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:21.667924    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8bvz" podUID="a140ffcb-b59c-4e55-95e8-e9ed589396b1"
Sep 10 15:42:23.666819 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.666486    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:23.667374 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.666487    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:23.667374 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:23.666850    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5l6v" podUID="f5ced900-9635-4cf4-9390-7396c3ef86a7"
Sep 10 15:42:23.667374 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:23.666950    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-znbph" podUID="8f29f7a6-d451-487d-92f1-84c47bf32d5c"
Sep 10 15:42:23.667374 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.666486    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:23.667374 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:23.667036    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8bvz" podUID="a140ffcb-b59c-4e55-95e8-e9ed589396b1"
Sep 10 15:42:23.735044 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.735015    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pl75t" event={"ID":"e1601814-5e66-4f97-8165-920af19fb9a4","Type":"ContainerStarted","Data":"c244addcf9e91df4924f25bdfc6f7e2d8ecbb36e98ec35fb5a0cb74a7bbcaa61"}
Sep 10 15:42:23.736581 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.736554    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-66dx8" event={"ID":"9cde9140-e403-43b0-bbda-d78cd59ce6d1","Type":"ContainerStarted","Data":"0a68df68977d5c3e57dc2e80d9dbe014642cd89a8c75b6463ad9c801e11f9945"}
Sep 10 15:42:23.737998 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.737976    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k6fhq" event={"ID":"e6f2f68d-170c-4567-ba06-b05bea2cb642","Type":"ContainerStarted","Data":"52cc5a412975bf8a949c180b51d7f6e13e49e0c4dbe60907e4febeaa07d0166a"}
Sep 10 15:42:23.739534 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.739513    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh" event={"ID":"4e0052e2-036e-47d2-9881-5d69873ddff4","Type":"ContainerStarted","Data":"58960b71873f045a027edbd5c93f6a35362f3aa4403e42713dd6d764931b3dea"}
Sep 10 15:42:23.742327 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.742308    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm" event={"ID":"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718","Type":"ContainerStarted","Data":"55653834c8f0ab6dcf6903a5a0c33bf25d80c959b78de0c54f946dc2f4012479"}
Sep 10 15:42:23.742417 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.742375    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm" event={"ID":"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718","Type":"ContainerStarted","Data":"44f5a019361081f3285247219bca34bea63629ce3a4783caceee0e42a839277c"}
Sep 10 15:42:23.742465 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.742426    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm" event={"ID":"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718","Type":"ContainerStarted","Data":"a40f63f81a465adc65996123b5234d44bd7ac955c75e3b97bdab6cd3b7edc7a6"}
Sep 10 15:42:23.742498 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.742477    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm" event={"ID":"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718","Type":"ContainerStarted","Data":"f9a4bacb77fbc49426dca4517a5b2ed40c4b4ee0671044fdc17b811de7ac5325"}
Sep 10 15:42:23.742535 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.742523    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm" event={"ID":"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718","Type":"ContainerStarted","Data":"a1b10800aabc19ae1f597fad407fd5d564949fb4eb2eda7b488941553a723965"}
Sep 10 15:42:23.742583 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.742558    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm" event={"ID":"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718","Type":"ContainerStarted","Data":"2d37b1d053b148f030b5e8a74a4ea947b36970951366cb5f917a3c3019c650cd"}
Sep 10 15:42:23.743748 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.743727    2564 generic.go:356] "Generic (PLEG): container finished" podID="8dcad2a033892fede0cdda710bab032c" containerID="f21eac0b9c6a8d727361a5ee3bc64168f9afb21d756eb2b6314d23a1c5f97e81" exitCode=0
Sep 10 15:42:23.743827 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.743795    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-145.ec2.internal" event={"ID":"8dcad2a033892fede0cdda710bab032c","Type":"ContainerDied","Data":"f21eac0b9c6a8d727361a5ee3bc64168f9afb21d756eb2b6314d23a1c5f97e81"}
Sep 10 15:42:23.745307 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.745285    2564 generic.go:356] "Generic (PLEG): container finished" podID="e30c4619-7a14-46c2-b2e8-f18c219cfa66" containerID="62993d6681ef3f4c95b2005370811d382f0c3b9a40d664a0e81c7b4e092875ec" exitCode=0
Sep 10 15:42:23.745396 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.745367    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bjjkl" event={"ID":"e30c4619-7a14-46c2-b2e8-f18c219cfa66","Type":"ContainerDied","Data":"62993d6681ef3f4c95b2005370811d382f0c3b9a40d664a0e81c7b4e092875ec"}
Sep 10 15:42:23.746830 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.746780    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q7c8n" event={"ID":"0cad1bb8-0f30-4ca9-ad4a-fbf4f70122fb","Type":"ContainerStarted","Data":"0fda383cf6fd5e7c9a5baa37840ed395a602ecb527dc22d08c0e638d999d03e0"}
Sep 10 15:42:23.749515 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.749480    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-145.ec2.internal" podStartSLOduration=20.749469181 podStartE2EDuration="20.749469181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:04.704297896 +0000 UTC m=+3.508617163" watchObservedRunningTime="2026-09-10 15:42:23.749469181 +0000 UTC m=+22.553788486"
Sep 10 15:42:23.749821 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.749793    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pl75t" podStartSLOduration=3.271243882 podStartE2EDuration="21.749781816s" totalImagesPullingTime="18.478537934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:23.749248767 +0000 UTC m=+22.553568027" watchObservedRunningTime="2026-09-10 15:42:23.749781816 +0000 UTC m=+22.554101111"
Sep 10 15:42:23.792576 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.792517    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k6fhq" podStartSLOduration=3.357271917 podStartE2EDuration="21.792497365s" totalImagesPullingTime="18.435225448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:23.776764451 +0000 UTC m=+22.581083713" watchObservedRunningTime="2026-09-10 15:42:23.792497365 +0000 UTC m=+22.596816627"
Sep 10 15:42:23.806716 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:23.806427    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-66dx8" podStartSLOduration=3.361325618 podStartE2EDuration="21.806407086s" totalImagesPullingTime="18.445081468s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:23.806402554 +0000 UTC m=+22.610721814" watchObservedRunningTime="2026-09-10 15:42:23.806407086 +0000 UTC m=+22.610726348"
Sep 10 15:42:24.318050 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:24.318020    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 15:42:24.640315 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:24.640152    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T15:42:24.318040531Z","UUID":"d279792f-722b-48df-93bc-7da97b4314c2","Handler":null,"Name":"","Endpoint":""}
Sep 10 15:42:24.641728 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:24.641701    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 15:42:24.641728 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:24.641729    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 15:42:24.751865 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:24.751829    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-145.ec2.internal" event={"ID":"8dcad2a033892fede0cdda710bab032c","Type":"ContainerStarted","Data":"eeca86619acddb11f7c0245cce00e286b959abd74b321522ad2c8e94113fb51e"}
Sep 10 15:42:24.754003 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:24.753579    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k5bnn" event={"ID":"24639f5d-eda2-4bf4-a4a4-112ec08b366a","Type":"ContainerStarted","Data":"ec5d94df5f351ef5b3a6c101789a598c74904718c63d1348f88509642c7c4f73"}
Sep 10 15:42:24.755790 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:24.755757    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh" event={"ID":"4e0052e2-036e-47d2-9881-5d69873ddff4","Type":"ContainerStarted","Data":"027e115c92a01629e2c4286c463991c79dad84eaae057c62aa15e51970abdba4"}
Sep 10 15:42:24.764987 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:24.764929    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q7c8n" podStartSLOduration=4.343700549 podStartE2EDuration="22.764913088s" totalImagesPullingTime="18.421212539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:23.819106347 +0000 UTC m=+22.623425608" watchObservedRunningTime="2026-09-10 15:42:24.764913088 +0000 UTC m=+23.569232351"
Sep 10 15:42:24.765188 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:24.765142    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-145.ec2.internal" podStartSLOduration=21.765132982 podStartE2EDuration="21.765132982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:24.76452269 +0000 UTC m=+23.568841953" watchObservedRunningTime="2026-09-10 15:42:24.765132982 +0000 UTC m=+23.569452244"
Sep 10 15:42:24.889233 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:24.889197    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q7c8n"
Sep 10 15:42:24.889892 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:24.889864    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q7c8n"
Sep 10 15:42:24.903556 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:24.903468    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k5bnn" podStartSLOduration=4.468363016 podStartE2EDuration="22.903452211s" totalImagesPullingTime="18.435089195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:24.775812707 +0000 UTC m=+23.580131973" watchObservedRunningTime="2026-09-10 15:42:24.903452211 +0000 UTC m=+23.707771473"
Sep 10 15:42:25.666904 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:25.666617    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:25.667024 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:25.666620    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:25.667024 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:25.666671    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:25.667131 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:25.667085    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8bvz" podUID="a140ffcb-b59c-4e55-95e8-e9ed589396b1"
Sep 10 15:42:25.667225 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:25.667206    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5l6v" podUID="f5ced900-9635-4cf4-9390-7396c3ef86a7"
Sep 10 15:42:25.667364 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:25.667335    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-znbph" podUID="8f29f7a6-d451-487d-92f1-84c47bf32d5c"
Sep 10 15:42:25.760093 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:25.760053    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh" event={"ID":"4e0052e2-036e-47d2-9881-5d69873ddff4","Type":"ContainerStarted","Data":"354e8e00706db0679540b87b4fc9d48b3fdbdff56e4c31194bff80ae7c52f15d"}
Sep 10 15:42:25.763250 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:25.763219    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm" event={"ID":"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718","Type":"ContainerStarted","Data":"28860e61e91ea4741edbde1fb955944e3eb0f9f0c477e51af56e0f5523b10679"}
Sep 10 15:42:25.763373 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:25.763341    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q7c8n"
Sep 10 15:42:25.763868 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:25.763847    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q7c8n"
Sep 10 15:42:25.775115 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:25.775071    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sglsh" podStartSLOduration=3.395036231 podStartE2EDuration="23.775057977s" totalImagesPullingTime="20.380021746s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:25.774535807 +0000 UTC m=+24.578855070" watchObservedRunningTime="2026-09-10 15:42:25.775057977 +0000 UTC m=+24.579377237"
Sep 10 15:42:27.666952 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:27.666490    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:27.666952 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:27.666491    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:27.666952 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:27.666622    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:27.666952 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:27.666628    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8bvz" podUID="a140ffcb-b59c-4e55-95e8-e9ed589396b1"
Sep 10 15:42:27.666952 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:27.666704    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5l6v" podUID="f5ced900-9635-4cf4-9390-7396c3ef86a7"
Sep 10 15:42:27.666952 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:27.666786    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-znbph" podUID="8f29f7a6-d451-487d-92f1-84c47bf32d5c"
Sep 10 15:42:28.772884 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:28.772820    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm" event={"ID":"2f3e30ed-505d-4f2b-a1a7-3033e1b1d718","Type":"ContainerStarted","Data":"fbb25a782e2798292bb691041f80f709be92473d75478da0499a232fa0e08b77"}
Sep 10 15:42:28.773265 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:28.773190    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:28.773329 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:28.773264    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:28.773329 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:28.773324    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:28.789689 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:28.789666    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:28.791460 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:28.791441    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:42:28.802452 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:28.802413    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm" podStartSLOduration=8.196957556 podStartE2EDuration="26.802401406s" totalImagesPullingTime="18.60544385s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:28.801872769 +0000 UTC m=+27.606192029" watchObservedRunningTime="2026-09-10 15:42:28.802401406 +0000 UTC m=+27.606720666"
Sep 10 15:42:29.669643 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:29.669453    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:29.669874 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:29.669487    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:29.669874 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:29.669744    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8bvz" podUID="a140ffcb-b59c-4e55-95e8-e9ed589396b1"
Sep 10 15:42:29.669874 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:29.669785    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5l6v" podUID="f5ced900-9635-4cf4-9390-7396c3ef86a7"
Sep 10 15:42:29.669874 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:29.669499    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:29.670002 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:29.669873    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-znbph" podUID="8f29f7a6-d451-487d-92f1-84c47bf32d5c"
Sep 10 15:42:29.781394 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:29.781350    2564 generic.go:356] "Generic (PLEG): container finished" podID="e30c4619-7a14-46c2-b2e8-f18c219cfa66" containerID="636afd5e10b7f85f8bbbb0fdbd9285616250f49df06a9fba51128499b348cc7d" exitCode=0
Sep 10 15:42:29.781753 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:29.781442    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bjjkl" event={"ID":"e30c4619-7a14-46c2-b2e8-f18c219cfa66","Type":"ContainerDied","Data":"636afd5e10b7f85f8bbbb0fdbd9285616250f49df06a9fba51128499b348cc7d"}
Sep 10 15:42:30.713902 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:30.713850    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-znbph"]
Sep 10 15:42:30.714119 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:30.713972    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:30.714119 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:30.714078    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-znbph" podUID="8f29f7a6-d451-487d-92f1-84c47bf32d5c"
Sep 10 15:42:30.716453 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:30.716428    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g5l6v"]
Sep 10 15:42:30.716558 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:30.716523    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:30.716629 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:30.716608    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5l6v" podUID="f5ced900-9635-4cf4-9390-7396c3ef86a7"
Sep 10 15:42:30.719440 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:30.719418    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m8bvz"]
Sep 10 15:42:30.719555 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:30.719499    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:30.719623 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:30.719603    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8bvz" podUID="a140ffcb-b59c-4e55-95e8-e9ed589396b1"
Sep 10 15:42:31.788000 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:31.787968    2564 generic.go:356] "Generic (PLEG): container finished" podID="e30c4619-7a14-46c2-b2e8-f18c219cfa66" containerID="2445aa909b348792dd35bd96a109872ad3ed32815fc9c8a9829db7f723e50afd" exitCode=0
Sep 10 15:42:31.788840 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:31.788016    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bjjkl" event={"ID":"e30c4619-7a14-46c2-b2e8-f18c219cfa66","Type":"ContainerDied","Data":"2445aa909b348792dd35bd96a109872ad3ed32815fc9c8a9829db7f723e50afd"}
Sep 10 15:42:32.666639 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:32.666465    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:32.666772 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:32.666485    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:32.666772 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:32.666718    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8bvz" podUID="a140ffcb-b59c-4e55-95e8-e9ed589396b1"
Sep 10 15:42:32.666856 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:32.666519    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:32.666856 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:32.666792    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5l6v" podUID="f5ced900-9635-4cf4-9390-7396c3ef86a7"
Sep 10 15:42:32.666925 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:32.666870    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-znbph" podUID="8f29f7a6-d451-487d-92f1-84c47bf32d5c"
Sep 10 15:42:32.792371 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:32.792338    2564 generic.go:356] "Generic (PLEG): container finished" podID="e30c4619-7a14-46c2-b2e8-f18c219cfa66" containerID="5101b564ea43387840fe47d5baf3f118e612870ebbfc749e4f7cf77a60e51201" exitCode=0
Sep 10 15:42:32.792706 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:32.792382    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bjjkl" event={"ID":"e30c4619-7a14-46c2-b2e8-f18c219cfa66","Type":"ContainerDied","Data":"5101b564ea43387840fe47d5baf3f118e612870ebbfc749e4f7cf77a60e51201"}
Sep 10 15:42:34.666485 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:34.666395    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:34.666956 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:34.666405    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:34.666956 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:34.666508    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m8bvz" podUID="a140ffcb-b59c-4e55-95e8-e9ed589396b1"
Sep 10 15:42:34.666956 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:34.666541    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:34.666956 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:34.666659    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-znbph" podUID="8f29f7a6-d451-487d-92f1-84c47bf32d5c"
Sep 10 15:42:34.666956 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:34.666732    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5l6v" podUID="f5ced900-9635-4cf4-9390-7396c3ef86a7"
Sep 10 15:42:35.326269 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:35.326229    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs\") pod \"network-metrics-daemon-m8bvz\" (UID: \"a140ffcb-b59c-4e55-95e8-e9ed589396b1\") " pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:35.326443 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:35.326377    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:35.326489 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:35.326452    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs podName:a140ffcb-b59c-4e55-95e8-e9ed589396b1 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:07.326433443 +0000 UTC m=+66.130752704 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs") pod "network-metrics-daemon-m8bvz" (UID: "a140ffcb-b59c-4e55-95e8-e9ed589396b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:35.427427 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:35.427393    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret\") pod \"global-pull-secret-syncer-znbph\" (UID: \"8f29f7a6-d451-487d-92f1-84c47bf32d5c\") " pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:35.427573 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:35.427500    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:35.427573 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:35.427556    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret podName:8f29f7a6-d451-487d-92f1-84c47bf32d5c nodeName:}" failed. No retries permitted until 2026-09-10 15:43:07.427541475 +0000 UTC m=+66.231860734 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret") pod "global-pull-secret-syncer-znbph" (UID: "8f29f7a6-d451-487d-92f1-84c47bf32d5c") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:35.528880 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:35.528278    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgz7t\" (UniqueName: \"kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t\") pod \"network-check-target-g5l6v\" (UID: \"f5ced900-9635-4cf4-9390-7396c3ef86a7\") " pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:35.528880 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:35.528416    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:42:35.528880 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:35.528440    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:42:35.528880 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:35.528455    2564 projected.go:196] Error preparing data for projected volume kube-api-access-dgz7t for pod openshift-network-diagnostics/network-check-target-g5l6v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:35.528880 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:35.528523    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t podName:f5ced900-9635-4cf4-9390-7396c3ef86a7 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:07.528503784 +0000 UTC m=+66.332823046 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dgz7t" (UniqueName: "kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t") pod "network-check-target-g5l6v" (UID: "f5ced900-9635-4cf4-9390-7396c3ef86a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:35.858273 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:35.858235    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-hb6s4"]
Sep 10 15:42:35.888149 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:35.888112    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hb6s4"
Sep 10 15:42:35.890693 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:35.890668    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 15:42:35.890693 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:35.890684    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kqss8\""
Sep 10 15:42:35.890941 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:35.890725    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 15:42:36.032859 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.032810    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mwxn\" (UniqueName: \"kubernetes.io/projected/c5fe8c6d-b8ce-4138-b5ed-a05499ccd346-kube-api-access-4mwxn\") pod \"node-resolver-hb6s4\" (UID: \"c5fe8c6d-b8ce-4138-b5ed-a05499ccd346\") " pod="openshift-dns/node-resolver-hb6s4"
Sep 10 15:42:36.033034 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.032982    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5fe8c6d-b8ce-4138-b5ed-a05499ccd346-tmp-dir\") pod \"node-resolver-hb6s4\" (UID: \"c5fe8c6d-b8ce-4138-b5ed-a05499ccd346\") " pod="openshift-dns/node-resolver-hb6s4"
Sep 10 15:42:36.033034 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.033029    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c5fe8c6d-b8ce-4138-b5ed-a05499ccd346-hosts-file\") pod \"node-resolver-hb6s4\" (UID: \"c5fe8c6d-b8ce-4138-b5ed-a05499ccd346\") " pod="openshift-dns/node-resolver-hb6s4"
Sep 10 15:42:36.133508 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.133414    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5fe8c6d-b8ce-4138-b5ed-a05499ccd346-tmp-dir\") pod \"node-resolver-hb6s4\" (UID: \"c5fe8c6d-b8ce-4138-b5ed-a05499ccd346\") " pod="openshift-dns/node-resolver-hb6s4"
Sep 10 15:42:36.133508 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.133470    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c5fe8c6d-b8ce-4138-b5ed-a05499ccd346-hosts-file\") pod \"node-resolver-hb6s4\" (UID: \"c5fe8c6d-b8ce-4138-b5ed-a05499ccd346\") " pod="openshift-dns/node-resolver-hb6s4"
Sep 10 15:42:36.133508 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.133507    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mwxn\" (UniqueName: \"kubernetes.io/projected/c5fe8c6d-b8ce-4138-b5ed-a05499ccd346-kube-api-access-4mwxn\") pod \"node-resolver-hb6s4\" (UID: \"c5fe8c6d-b8ce-4138-b5ed-a05499ccd346\") " pod="openshift-dns/node-resolver-hb6s4"
Sep 10 15:42:36.133783 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.133635    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c5fe8c6d-b8ce-4138-b5ed-a05499ccd346-hosts-file\") pod \"node-resolver-hb6s4\" (UID: \"c5fe8c6d-b8ce-4138-b5ed-a05499ccd346\") " pod="openshift-dns/node-resolver-hb6s4"
Sep 10 15:42:36.133838 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.133817    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5fe8c6d-b8ce-4138-b5ed-a05499ccd346-tmp-dir\") pod \"node-resolver-hb6s4\" (UID: \"c5fe8c6d-b8ce-4138-b5ed-a05499ccd346\") " pod="openshift-dns/node-resolver-hb6s4"
Sep 10 15:42:36.149840 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.149817    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mwxn\" (UniqueName: \"kubernetes.io/projected/c5fe8c6d-b8ce-4138-b5ed-a05499ccd346-kube-api-access-4mwxn\") pod \"node-resolver-hb6s4\" (UID: \"c5fe8c6d-b8ce-4138-b5ed-a05499ccd346\") " pod="openshift-dns/node-resolver-hb6s4"
Sep 10 15:42:36.198798 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.198755    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hb6s4"
Sep 10 15:42:36.519676 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.519645    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-145.ec2.internal" event="NodeReady"
Sep 10 15:42:36.519859 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.519794    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 15:42:36.570235 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.570199    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2qrbr"]
Sep 10 15:42:36.590308 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.590268    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8bc5c"]
Sep 10 15:42:36.590954 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.590939    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:42:36.593369 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.593347    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 15:42:36.593369 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.593347    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z4knc\""
Sep 10 15:42:36.593538 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.593347    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 15:42:36.604348 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.604325    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8bc5c"]
Sep 10 15:42:36.604475 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.604362    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2qrbr"]
Sep 10 15:42:36.604475 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.604468    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8bc5c"
Sep 10 15:42:36.607503 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.607477    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 15:42:36.607750 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.607734    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 15:42:36.607837 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.607822    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8f2wr\""
Sep 10 15:42:36.607947 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.607934    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 15:42:36.666293 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.666261    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:42:36.666487 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.666261    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:42:36.666555 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.666267    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:42:36.668865 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.668844    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 15:42:36.668999 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.668957    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cthzr\""
Sep 10 15:42:36.668999 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.668851    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 15:42:36.669087 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.669014    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 15:42:36.669211 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.669194    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 15:42:36.669292 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.669269    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b79h8\""
Sep 10 15:42:36.737789 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.737755    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thkt8\" (UniqueName: \"kubernetes.io/projected/28343b32-70a8-4d77-8dfd-d7b27a3f8cef-kube-api-access-thkt8\") pod \"ingress-canary-8bc5c\" (UID: \"28343b32-70a8-4d77-8dfd-d7b27a3f8cef\") " pod="openshift-ingress-canary/ingress-canary-8bc5c"
Sep 10 15:42:36.737789 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.737802    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28343b32-70a8-4d77-8dfd-d7b27a3f8cef-cert\") pod \"ingress-canary-8bc5c\" (UID: \"28343b32-70a8-4d77-8dfd-d7b27a3f8cef\") " pod="openshift-ingress-canary/ingress-canary-8bc5c"
Sep 10 15:42:36.738034 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.737899    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/db9487f8-8adc-4d2e-8c5b-7222bf915898-config-volume\") pod \"dns-default-2qrbr\" (UID: \"db9487f8-8adc-4d2e-8c5b-7222bf915898\") " pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:42:36.738034 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.737928    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls\") pod \"dns-default-2qrbr\" (UID: \"db9487f8-8adc-4d2e-8c5b-7222bf915898\") " pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:42:36.738034 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.737949    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db9487f8-8adc-4d2e-8c5b-7222bf915898-tmp-dir\") pod \"dns-default-2qrbr\" (UID: \"db9487f8-8adc-4d2e-8c5b-7222bf915898\") " pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:42:36.738187 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.738030    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whp2p\" (UniqueName: \"kubernetes.io/projected/db9487f8-8adc-4d2e-8c5b-7222bf915898-kube-api-access-whp2p\") pod \"dns-default-2qrbr\" (UID: \"db9487f8-8adc-4d2e-8c5b-7222bf915898\") " pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:42:36.838463 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.838369    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whp2p\" (UniqueName: \"kubernetes.io/projected/db9487f8-8adc-4d2e-8c5b-7222bf915898-kube-api-access-whp2p\") pod \"dns-default-2qrbr\" (UID: \"db9487f8-8adc-4d2e-8c5b-7222bf915898\") " pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:42:36.838463 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.838438    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thkt8\" (UniqueName: \"kubernetes.io/projected/28343b32-70a8-4d77-8dfd-d7b27a3f8cef-kube-api-access-thkt8\") pod \"ingress-canary-8bc5c\" (UID: \"28343b32-70a8-4d77-8dfd-d7b27a3f8cef\") " pod="openshift-ingress-canary/ingress-canary-8bc5c"
Sep 10 15:42:36.838686 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.838467    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28343b32-70a8-4d77-8dfd-d7b27a3f8cef-cert\") pod \"ingress-canary-8bc5c\" (UID: \"28343b32-70a8-4d77-8dfd-d7b27a3f8cef\") " pod="openshift-ingress-canary/ingress-canary-8bc5c"
Sep 10 15:42:36.838745 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.838711    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/db9487f8-8adc-4d2e-8c5b-7222bf915898-config-volume\") pod \"dns-default-2qrbr\" (UID: \"db9487f8-8adc-4d2e-8c5b-7222bf915898\") " pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:42:36.838803 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.838746    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls\") pod \"dns-default-2qrbr\" (UID: \"db9487f8-8adc-4d2e-8c5b-7222bf915898\") " pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:42:36.838803 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.838779    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db9487f8-8adc-4d2e-8c5b-7222bf915898-tmp-dir\") pod \"dns-default-2qrbr\" (UID: \"db9487f8-8adc-4d2e-8c5b-7222bf915898\") " pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:42:36.838904 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:36.838884    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:42:36.838971 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:36.838957    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls podName:db9487f8-8adc-4d2e-8c5b-7222bf915898 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:37.338936665 +0000 UTC m=+36.143255926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls") pod "dns-default-2qrbr" (UID: "db9487f8-8adc-4d2e-8c5b-7222bf915898") : secret "dns-default-metrics-tls" not found
Sep 10 15:42:36.839111 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.839094    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db9487f8-8adc-4d2e-8c5b-7222bf915898-tmp-dir\") pod \"dns-default-2qrbr\" (UID: \"db9487f8-8adc-4d2e-8c5b-7222bf915898\") " pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:42:36.839325 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.839303    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/db9487f8-8adc-4d2e-8c5b-7222bf915898-config-volume\") pod \"dns-default-2qrbr\" (UID: \"db9487f8-8adc-4d2e-8c5b-7222bf915898\") " pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:42:36.841322 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.841299    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28343b32-70a8-4d77-8dfd-d7b27a3f8cef-cert\") pod \"ingress-canary-8bc5c\" (UID: \"28343b32-70a8-4d77-8dfd-d7b27a3f8cef\") " pod="openshift-ingress-canary/ingress-canary-8bc5c"
Sep 10 15:42:36.851002 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.850976    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thkt8\" (UniqueName: \"kubernetes.io/projected/28343b32-70a8-4d77-8dfd-d7b27a3f8cef-kube-api-access-thkt8\") pod \"ingress-canary-8bc5c\" (UID: \"28343b32-70a8-4d77-8dfd-d7b27a3f8cef\") " pod="openshift-ingress-canary/ingress-canary-8bc5c"
Sep 10 15:42:36.851126 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.850979    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whp2p\" (UniqueName: \"kubernetes.io/projected/db9487f8-8adc-4d2e-8c5b-7222bf915898-kube-api-access-whp2p\") pod \"dns-default-2qrbr\" (UID: \"db9487f8-8adc-4d2e-8c5b-7222bf915898\") " pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:42:36.914605 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:36.914568    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8bc5c"
Sep 10 15:42:37.343379 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:37.343349    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5fe8c6d_b8ce_4138_b5ed_a05499ccd346.slice/crio-0d4eb4ba8fd4bcb3e4065fc9117ac3d5aed71b0dfa0829fdd942d83370b66274 WatchSource:0}: Error finding container 0d4eb4ba8fd4bcb3e4065fc9117ac3d5aed71b0dfa0829fdd942d83370b66274: Status 404 returned error can't find the container with id 0d4eb4ba8fd4bcb3e4065fc9117ac3d5aed71b0dfa0829fdd942d83370b66274
Sep 10 15:42:37.343467 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:37.343454    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls\") pod \"dns-default-2qrbr\" (UID: \"db9487f8-8adc-4d2e-8c5b-7222bf915898\") " pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:42:37.343623 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:37.343611    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:42:37.343678 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:37.343668    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls podName:db9487f8-8adc-4d2e-8c5b-7222bf915898 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:38.343652268 +0000 UTC m=+37.147971527 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls") pod "dns-default-2qrbr" (UID: "db9487f8-8adc-4d2e-8c5b-7222bf915898") : secret "dns-default-metrics-tls" not found
Sep 10 15:42:37.631644 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:37.630052    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8bc5c"]
Sep 10 15:42:37.634830 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:42:37.634799    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28343b32_70a8_4d77_8dfd_d7b27a3f8cef.slice/crio-1ffe6d931b54b165787996db6bac183a38dc9f84d938f4c4673bef264c623727 WatchSource:0}: Error finding container 1ffe6d931b54b165787996db6bac183a38dc9f84d938f4c4673bef264c623727: Status 404 returned error can't find the container with id 1ffe6d931b54b165787996db6bac183a38dc9f84d938f4c4673bef264c623727
Sep 10 15:42:37.806985 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:37.806956    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hb6s4" event={"ID":"c5fe8c6d-b8ce-4138-b5ed-a05499ccd346","Type":"ContainerStarted","Data":"0b74c0c6a2245a83c5f094f390264a951b0eacdf892e56cde176e61fc2f11afb"}
Sep 10 15:42:37.807117 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:37.807007    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hb6s4" event={"ID":"c5fe8c6d-b8ce-4138-b5ed-a05499ccd346","Type":"ContainerStarted","Data":"0d4eb4ba8fd4bcb3e4065fc9117ac3d5aed71b0dfa0829fdd942d83370b66274"}
Sep 10 15:42:37.809730 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:37.809708    2564 generic.go:356] "Generic (PLEG): container finished" podID="e30c4619-7a14-46c2-b2e8-f18c219cfa66" containerID="0179322c5f921c4cecc5841135c38c341368aa1a22359ec8b6818d15e7a7614c" exitCode=0
Sep 10 15:42:37.809842 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:37.809792    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bjjkl" event={"ID":"e30c4619-7a14-46c2-b2e8-f18c219cfa66","Type":"ContainerDied","Data":"0179322c5f921c4cecc5841135c38c341368aa1a22359ec8b6818d15e7a7614c"}
Sep 10 15:42:37.811051 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:37.811025    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8bc5c" event={"ID":"28343b32-70a8-4d77-8dfd-d7b27a3f8cef","Type":"ContainerStarted","Data":"1ffe6d931b54b165787996db6bac183a38dc9f84d938f4c4673bef264c623727"}
Sep 10 15:42:37.822917 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:37.822862    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hb6s4" podStartSLOduration=2.8228438369999997 podStartE2EDuration="2.822843837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:37.821684312 +0000 UTC m=+36.626003574" watchObservedRunningTime="2026-09-10 15:42:37.822843837 +0000 UTC m=+36.627163099"
Sep 10 15:42:38.349238 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:38.349206    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls\") pod \"dns-default-2qrbr\" (UID: \"db9487f8-8adc-4d2e-8c5b-7222bf915898\") " pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:42:38.349878 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:38.349367    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:42:38.349878 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:38.349434    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls podName:db9487f8-8adc-4d2e-8c5b-7222bf915898 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:40.34941732 +0000 UTC m=+39.153736560 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls") pod "dns-default-2qrbr" (UID: "db9487f8-8adc-4d2e-8c5b-7222bf915898") : secret "dns-default-metrics-tls" not found
Sep 10 15:42:38.818229 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:38.818192    2564 generic.go:356] "Generic (PLEG): container finished" podID="e30c4619-7a14-46c2-b2e8-f18c219cfa66" containerID="ebf0f0c20579e91f43f51cff9a982ab59e62bb05f739f288ad336852170ddfd4" exitCode=0
Sep 10 15:42:38.818414 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:38.818273    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bjjkl" event={"ID":"e30c4619-7a14-46c2-b2e8-f18c219cfa66","Type":"ContainerDied","Data":"ebf0f0c20579e91f43f51cff9a982ab59e62bb05f739f288ad336852170ddfd4"}
Sep 10 15:42:39.824892 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:39.824866    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bjjkl" event={"ID":"e30c4619-7a14-46c2-b2e8-f18c219cfa66","Type":"ContainerStarted","Data":"e887fa325cdaade3368776fee3f6f98140d9ad3e58ab9270d0cb10fe774c62b1"}
Sep 10 15:42:39.850093 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:39.850047    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bjjkl" podStartSLOduration=8.149989554 podStartE2EDuration="37.850030475s" totalImagesPullingTime="29.700040921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:39.848305007 +0000 UTC m=+38.652624293" watchObservedRunningTime="2026-09-10 15:42:39.850030475 +0000 UTC m=+38.654349737"
Sep 10 15:42:40.365554 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:40.365419    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls\") pod \"dns-default-2qrbr\" (UID: \"db9487f8-8adc-4d2e-8c5b-7222bf915898\") " pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:42:40.365665 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:40.365580    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:42:40.365665 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:40.365647    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls podName:db9487f8-8adc-4d2e-8c5b-7222bf915898 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:44.365629707 +0000 UTC m=+43.169948965 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls") pod "dns-default-2qrbr" (UID: "db9487f8-8adc-4d2e-8c5b-7222bf915898") : secret "dns-default-metrics-tls" not found
Sep 10 15:42:40.828476 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:40.828439    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8bc5c" event={"ID":"28343b32-70a8-4d77-8dfd-d7b27a3f8cef","Type":"ContainerStarted","Data":"769b5472fc16cb36d899412c033f976a8c05dcaf9b5887ac36b2c4690e3a7f7f"}
Sep 10 15:42:40.845457 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:40.845386    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8bc5c" podStartSLOduration=2.157603682 podStartE2EDuration="4.845369297s" totalImagesPullingTime="2.687765615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:40.844880098 +0000 UTC m=+39.649199357" watchObservedRunningTime="2026-09-10 15:42:40.845369297 +0000 UTC m=+39.649688558"
Sep 10 15:42:44.390678 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:44.390637    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls\") pod \"dns-default-2qrbr\" (UID: \"db9487f8-8adc-4d2e-8c5b-7222bf915898\") " pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:42:44.391044 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:44.390749    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:42:44.391044 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:44.390799    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls podName:db9487f8-8adc-4d2e-8c5b-7222bf915898 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:52.390786413 +0000 UTC m=+51.195105652 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls") pod "dns-default-2qrbr" (UID: "db9487f8-8adc-4d2e-8c5b-7222bf915898") : secret "dns-default-metrics-tls" not found
Sep 10 15:42:52.445084 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:42:52.445045    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls\") pod \"dns-default-2qrbr\" (UID: \"db9487f8-8adc-4d2e-8c5b-7222bf915898\") " pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:42:52.445556 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:52.445192    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:42:52.445556 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:42:52.445261    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls podName:db9487f8-8adc-4d2e-8c5b-7222bf915898 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:08.445243269 +0000 UTC m=+67.249562526 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls") pod "dns-default-2qrbr" (UID: "db9487f8-8adc-4d2e-8c5b-7222bf915898") : secret "dns-default-metrics-tls" not found
Sep 10 15:43:00.794741 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:00.794709    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4xzlm"
Sep 10 15:43:07.354616 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:07.354559    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs\") pod \"network-metrics-daemon-m8bvz\" (UID: \"a140ffcb-b59c-4e55-95e8-e9ed589396b1\") " pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:43:07.357386 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:07.357357    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 15:43:07.365674 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:07.365655    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 15:43:07.365736 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:07.365725    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs podName:a140ffcb-b59c-4e55-95e8-e9ed589396b1 nodeName:}" failed. No retries permitted until 2026-09-10 15:44:11.365708614 +0000 UTC m=+130.170027853 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs") pod "network-metrics-daemon-m8bvz" (UID: "a140ffcb-b59c-4e55-95e8-e9ed589396b1") : secret "metrics-daemon-secret" not found
Sep 10 15:43:07.455567 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:07.455533    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret\") pod \"global-pull-secret-syncer-znbph\" (UID: \"8f29f7a6-d451-487d-92f1-84c47bf32d5c\") " pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:43:07.458168 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:07.458136    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 15:43:07.469728 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:07.469704    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f29f7a6-d451-487d-92f1-84c47bf32d5c-original-pull-secret\") pod \"global-pull-secret-syncer-znbph\" (UID: \"8f29f7a6-d451-487d-92f1-84c47bf32d5c\") " pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:43:07.556437 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:07.556378    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgz7t\" (UniqueName: \"kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t\") pod \"network-check-target-g5l6v\" (UID: \"f5ced900-9635-4cf4-9390-7396c3ef86a7\") " pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:43:07.559626 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:07.559604    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 15:43:07.569646 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:07.569623    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 15:43:07.581018 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:07.580995    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgz7t\" (UniqueName: \"kubernetes.io/projected/f5ced900-9635-4cf4-9390-7396c3ef86a7-kube-api-access-dgz7t\") pod \"network-check-target-g5l6v\" (UID: \"f5ced900-9635-4cf4-9390-7396c3ef86a7\") " pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:43:07.590034 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:07.590017    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-znbph"
Sep 10 15:43:07.857395 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:07.857365    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-znbph"]
Sep 10 15:43:07.860501 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:43:07.860471    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f29f7a6_d451_487d_92f1_84c47bf32d5c.slice/crio-2047ed8c23cca5387d73cff6c10e0ee3e503a0d6270c9f132061dc6b3c458dcb WatchSource:0}: Error finding container 2047ed8c23cca5387d73cff6c10e0ee3e503a0d6270c9f132061dc6b3c458dcb: Status 404 returned error can't find the container with id 2047ed8c23cca5387d73cff6c10e0ee3e503a0d6270c9f132061dc6b3c458dcb
Sep 10 15:43:07.880392 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:07.880371    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cthzr\""
Sep 10 15:43:07.888659 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:07.888607    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:43:07.893677 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:07.893655    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-znbph" event={"ID":"8f29f7a6-d451-487d-92f1-84c47bf32d5c","Type":"ContainerStarted","Data":"2047ed8c23cca5387d73cff6c10e0ee3e503a0d6270c9f132061dc6b3c458dcb"}
Sep 10 15:43:08.156738 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.156647    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g5l6v"]
Sep 10 15:43:08.160382 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:43:08.160356    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ced900_9635_4cf4_9390_7396c3ef86a7.slice/crio-19f1aecc42e639757091921d23845087d81db19f2efbca8383c5a1e20ac418e1 WatchSource:0}: Error finding container 19f1aecc42e639757091921d23845087d81db19f2efbca8383c5a1e20ac418e1: Status 404 returned error can't find the container with id 19f1aecc42e639757091921d23845087d81db19f2efbca8383c5a1e20ac418e1
Sep 10 15:43:08.462197 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.462136    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls\") pod \"dns-default-2qrbr\" (UID: \"db9487f8-8adc-4d2e-8c5b-7222bf915898\") " pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:43:08.462597 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:08.462299    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:43:08.462597 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:08.462377    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls podName:db9487f8-8adc-4d2e-8c5b-7222bf915898 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:40.462360553 +0000 UTC m=+99.266679814 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls") pod "dns-default-2qrbr" (UID: "db9487f8-8adc-4d2e-8c5b-7222bf915898") : secret "dns-default-metrics-tls" not found
Sep 10 15:43:08.787919 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.786853    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-g952v"]
Sep 10 15:43:08.789044 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.789012    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-g952v"
Sep 10 15:43:08.792305 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.791754    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 10 15:43:08.792305 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.791971    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zzfht\""
Sep 10 15:43:08.793073 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.792801    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:43:08.793444 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.793263    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 10 15:43:08.807771 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.807706    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-g952v"]
Sep 10 15:43:08.865240 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.865203    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkd2t\" (UniqueName: \"kubernetes.io/projected/1c3c7c51-bd5e-4508-8998-804694391874-kube-api-access-mkd2t\") pod \"cluster-samples-operator-7cdcfcf76c-g952v\" (UID: \"1c3c7c51-bd5e-4508-8998-804694391874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-g952v"
Sep 10 15:43:08.865413 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.865250    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c3c7c51-bd5e-4508-8998-804694391874-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-g952v\" (UID: \"1c3c7c51-bd5e-4508-8998-804694391874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-g952v"
Sep 10 15:43:08.888719 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.888688    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-snm4z"]
Sep 10 15:43:08.890833 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.890812    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-snm4z"
Sep 10 15:43:08.892664 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.892362    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r"]
Sep 10 15:43:08.893605 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.893576    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 10 15:43:08.894610 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.893576    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:43:08.894610 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.893634    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-28dz2\""
Sep 10 15:43:08.894610 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.893855    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 10 15:43:08.894610 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.893933    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 10 15:43:08.894610 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.894424    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-g45nr"]
Sep 10 15:43:08.895078 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.895055    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r"
Sep 10 15:43:08.896312 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.896292    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-g45nr"
Sep 10 15:43:08.898460 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.898003    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8dn2n\""
Sep 10 15:43:08.898460 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.898007    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 15:43:08.898460 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.898278    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 15:43:08.898460 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.898293    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g5l6v" event={"ID":"f5ced900-9635-4cf4-9390-7396c3ef86a7","Type":"ContainerStarted","Data":"19f1aecc42e639757091921d23845087d81db19f2efbca8383c5a1e20ac418e1"}
Sep 10 15:43:08.899128 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.899107    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 10 15:43:08.900289 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.900268    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 10 15:43:08.900384 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.900336    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 10 15:43:08.900547 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.900530    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:43:08.900634 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.900619    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 10 15:43:08.900758 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.900742    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-gbshg\""
Sep 10 15:43:08.902652 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.902632    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-snm4z"]
Sep 10 15:43:08.903883 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.903838    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-g45nr"]
Sep 10 15:43:08.919758 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.919735    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r"]
Sep 10 15:43:08.966303 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.966268    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f-serving-cert\") pod \"console-operator-85b454874b-snm4z\" (UID: \"f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f\") " pod="openshift-console-operator/console-operator-85b454874b-snm4z"
Sep 10 15:43:08.966480 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.966346    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkd2t\" (UniqueName: \"kubernetes.io/projected/1c3c7c51-bd5e-4508-8998-804694391874-kube-api-access-mkd2t\") pod \"cluster-samples-operator-7cdcfcf76c-g952v\" (UID: \"1c3c7c51-bd5e-4508-8998-804694391874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-g952v"
Sep 10 15:43:08.966480 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.966378    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c3c7c51-bd5e-4508-8998-804694391874-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-g952v\" (UID: \"1c3c7c51-bd5e-4508-8998-804694391874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-g952v"
Sep 10 15:43:08.966480 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.966406    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f-trusted-ca\") pod \"console-operator-85b454874b-snm4z\" (UID: \"f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f\") " pod="openshift-console-operator/console-operator-85b454874b-snm4z"
Sep 10 15:43:08.966480 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.966431    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr92j\" (UniqueName: \"kubernetes.io/projected/f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f-kube-api-access-lr92j\") pod \"console-operator-85b454874b-snm4z\" (UID: \"f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f\") " pod="openshift-console-operator/console-operator-85b454874b-snm4z"
Sep 10 15:43:08.966480 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.966454    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fba202c6-588a-4976-b518-e1e95ae87c61-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-6qs4r\" (UID: \"fba202c6-588a-4976-b518-e1e95ae87c61\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r"
Sep 10 15:43:08.966805 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:08.966520    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 15:43:08.966805 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.966538    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2k67\" (UniqueName: \"kubernetes.io/projected/fba202c6-588a-4976-b518-e1e95ae87c61-kube-api-access-g2k67\") pod \"cluster-monitoring-operator-6cdcc96965-6qs4r\" (UID: \"fba202c6-588a-4976-b518-e1e95ae87c61\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r"
Sep 10 15:43:08.966805 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:08.966588    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c3c7c51-bd5e-4508-8998-804694391874-samples-operator-tls podName:1c3c7c51-bd5e-4508-8998-804694391874 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:09.466572478 +0000 UTC m=+68.270891740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c3c7c51-bd5e-4508-8998-804694391874-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-g952v" (UID: "1c3c7c51-bd5e-4508-8998-804694391874") : secret "samples-operator-tls" not found
Sep 10 15:43:08.966805 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.966606    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxlzn\" (UniqueName: \"kubernetes.io/projected/14920648-4049-47e8-983a-8b9d7f41182b-kube-api-access-xxlzn\") pod \"volume-data-source-validator-695c895bd6-g45nr\" (UID: \"14920648-4049-47e8-983a-8b9d7f41182b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-g45nr"
Sep 10 15:43:08.966805 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.966656    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fba202c6-588a-4976-b518-e1e95ae87c61-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6qs4r\" (UID: \"fba202c6-588a-4976-b518-e1e95ae87c61\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r"
Sep 10 15:43:08.966805 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.966723    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f-config\") pod \"console-operator-85b454874b-snm4z\" (UID: \"f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f\") " pod="openshift-console-operator/console-operator-85b454874b-snm4z"
Sep 10 15:43:08.975372 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.975345    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkd2t\" (UniqueName: \"kubernetes.io/projected/1c3c7c51-bd5e-4508-8998-804694391874-kube-api-access-mkd2t\") pod \"cluster-samples-operator-7cdcfcf76c-g952v\" (UID: \"1c3c7c51-bd5e-4508-8998-804694391874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-g952v"
Sep 10 15:43:08.990335 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.990309    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-nsnkp"]
Sep 10 15:43:08.992482 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.992459    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-kk7mn"]
Sep 10 15:43:08.992640 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.992559    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-nsnkp"
Sep 10 15:43:08.994428 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.994408    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn"
Sep 10 15:43:08.995218 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.995060    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 10 15:43:08.995218 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.995073    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 10 15:43:08.995379 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.995276    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 10 15:43:08.995379 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.995321    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:43:08.995481 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.995411    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-54bbl\""
Sep 10 15:43:08.997180 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.996881    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 15:43:08.997684 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.997656    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 10 15:43:08.998364 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.998342    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hhz4t\""
Sep 10 15:43:09.000512 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.998843    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 15:43:09.000512 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:08.999180    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 10 15:43:09.003698 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.003003    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-nsnkp"]
Sep 10 15:43:09.003761 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.003699    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-kk7mn"]
Sep 10 15:43:09.006180 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.006140    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 10 15:43:09.067828 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.067746    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21f9ed82-8d7f-48dc-af6f-a22d73178304-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-kk7mn\" (UID: \"21f9ed82-8d7f-48dc-af6f-a22d73178304\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn"
Sep 10 15:43:09.067828 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.067799    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxlzn\" (UniqueName: \"kubernetes.io/projected/14920648-4049-47e8-983a-8b9d7f41182b-kube-api-access-xxlzn\") pod \"volume-data-source-validator-695c895bd6-g45nr\" (UID: \"14920648-4049-47e8-983a-8b9d7f41182b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-g45nr"
Sep 10 15:43:09.068024 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.067830    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/21f9ed82-8d7f-48dc-af6f-a22d73178304-snapshots\") pod \"insights-operator-76bcd6ccf5-kk7mn\" (UID: \"21f9ed82-8d7f-48dc-af6f-a22d73178304\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn"
Sep 10 15:43:09.068024 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.067880    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fba202c6-588a-4976-b518-e1e95ae87c61-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6qs4r\" (UID: \"fba202c6-588a-4976-b518-e1e95ae87c61\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r"
Sep 10 15:43:09.068024 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.067908    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f-config\") pod \"console-operator-85b454874b-snm4z\" (UID: \"f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f\") " pod="openshift-console-operator/console-operator-85b454874b-snm4z"
Sep 10 15:43:09.068024 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.067933    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w86n\" (UniqueName: \"kubernetes.io/projected/21f9ed82-8d7f-48dc-af6f-a22d73178304-kube-api-access-8w86n\") pod \"insights-operator-76bcd6ccf5-kk7mn\" (UID: \"21f9ed82-8d7f-48dc-af6f-a22d73178304\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn"
Sep 10 15:43:09.068024 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.067965    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f-serving-cert\") pod \"console-operator-85b454874b-snm4z\" (UID: \"f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f\") " pod="openshift-console-operator/console-operator-85b454874b-snm4z"
Sep 10 15:43:09.068024 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.067989    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/811b2e5a-e4b6-45d9-99d4-4669a476afbc-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-nsnkp\" (UID: \"811b2e5a-e4b6-45d9-99d4-4669a476afbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-nsnkp"
Sep 10 15:43:09.068024 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.068012    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21f9ed82-8d7f-48dc-af6f-a22d73178304-tmp\") pod \"insights-operator-76bcd6ccf5-kk7mn\" (UID: \"21f9ed82-8d7f-48dc-af6f-a22d73178304\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn"
Sep 10 15:43:09.068311 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.068064    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f-trusted-ca\") pod \"console-operator-85b454874b-snm4z\" (UID: \"f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f\") " pod="openshift-console-operator/console-operator-85b454874b-snm4z"
Sep 10 15:43:09.068311 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.068084    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr92j\" (UniqueName: \"kubernetes.io/projected/f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f-kube-api-access-lr92j\") pod \"console-operator-85b454874b-snm4z\" (UID: \"f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f\") " pod="openshift-console-operator/console-operator-85b454874b-snm4z"
Sep 10 15:43:09.068311 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.068108    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fba202c6-588a-4976-b518-e1e95ae87c61-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-6qs4r\" (UID: \"fba202c6-588a-4976-b518-e1e95ae87c61\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r"
Sep 10 15:43:09.068311 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.068133    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21f9ed82-8d7f-48dc-af6f-a22d73178304-serving-cert\") pod \"insights-operator-76bcd6ccf5-kk7mn\" (UID: \"21f9ed82-8d7f-48dc-af6f-a22d73178304\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn"
Sep 10 15:43:09.068311 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.068179    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/811b2e5a-e4b6-45d9-99d4-4669a476afbc-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-nsnkp\" (UID: \"811b2e5a-e4b6-45d9-99d4-4669a476afbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-nsnkp"
Sep 10 15:43:09.068311 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.068199    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21f9ed82-8d7f-48dc-af6f-a22d73178304-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-kk7mn\" (UID: \"21f9ed82-8d7f-48dc-af6f-a22d73178304\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn"
Sep 10 15:43:09.068311 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.068219    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5c6t\" (UniqueName: \"kubernetes.io/projected/811b2e5a-e4b6-45d9-99d4-4669a476afbc-kube-api-access-z5c6t\") pod \"kube-storage-version-migrator-operator-8d7dd4675-nsnkp\" (UID: \"811b2e5a-e4b6-45d9-99d4-4669a476afbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-nsnkp"
Sep 10 15:43:09.068311 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.068247    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2k67\" (UniqueName: \"kubernetes.io/projected/fba202c6-588a-4976-b518-e1e95ae87c61-kube-api-access-g2k67\") pod \"cluster-monitoring-operator-6cdcc96965-6qs4r\" (UID: \"fba202c6-588a-4976-b518-e1e95ae87c61\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r"
Sep 10 15:43:09.068750 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:09.068734    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 15:43:09.068800 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:09.068789    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fba202c6-588a-4976-b518-e1e95ae87c61-cluster-monitoring-operator-tls podName:fba202c6-588a-4976-b518-e1e95ae87c61 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:09.568772156 +0000 UTC m=+68.373091395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fba202c6-588a-4976-b518-e1e95ae87c61-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6qs4r" (UID: "fba202c6-588a-4976-b518-e1e95ae87c61") : secret "cluster-monitoring-operator-tls" not found
Sep 10 15:43:09.069397 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.069374    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f-config\") pod \"console-operator-85b454874b-snm4z\" (UID: \"f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f\") " pod="openshift-console-operator/console-operator-85b454874b-snm4z"
Sep 10 15:43:09.070662 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.070635    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fba202c6-588a-4976-b518-e1e95ae87c61-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-6qs4r\" (UID: \"fba202c6-588a-4976-b518-e1e95ae87c61\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r"
Sep 10 15:43:09.070787 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.070639    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f-trusted-ca\") pod \"console-operator-85b454874b-snm4z\" (UID: \"f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f\") " pod="openshift-console-operator/console-operator-85b454874b-snm4z"
Sep 10 15:43:09.072656 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.072635    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f-serving-cert\") pod \"console-operator-85b454874b-snm4z\" (UID: \"f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f\") " pod="openshift-console-operator/console-operator-85b454874b-snm4z"
Sep 10 15:43:09.077529 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.077502    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2k67\" (UniqueName: \"kubernetes.io/projected/fba202c6-588a-4976-b518-e1e95ae87c61-kube-api-access-g2k67\") pod \"cluster-monitoring-operator-6cdcc96965-6qs4r\" (UID: \"fba202c6-588a-4976-b518-e1e95ae87c61\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r"
Sep 10 15:43:09.077758 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.077736    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxlzn\" (UniqueName: \"kubernetes.io/projected/14920648-4049-47e8-983a-8b9d7f41182b-kube-api-access-xxlzn\") pod \"volume-data-source-validator-695c895bd6-g45nr\" (UID: \"14920648-4049-47e8-983a-8b9d7f41182b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-g45nr"
Sep 10 15:43:09.078691 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.078653    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr92j\" (UniqueName: \"kubernetes.io/projected/f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f-kube-api-access-lr92j\") pod \"console-operator-85b454874b-snm4z\" (UID: \"f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f\") " pod="openshift-console-operator/console-operator-85b454874b-snm4z"
Sep 10 15:43:09.168929 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.168886    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w86n\" (UniqueName: \"kubernetes.io/projected/21f9ed82-8d7f-48dc-af6f-a22d73178304-kube-api-access-8w86n\") pod \"insights-operator-76bcd6ccf5-kk7mn\" (UID: \"21f9ed82-8d7f-48dc-af6f-a22d73178304\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn"
Sep 10 15:43:09.169111 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.168946    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/811b2e5a-e4b6-45d9-99d4-4669a476afbc-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-nsnkp\" (UID: \"811b2e5a-e4b6-45d9-99d4-4669a476afbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-nsnkp"
Sep 10 15:43:09.169111 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.168975    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21f9ed82-8d7f-48dc-af6f-a22d73178304-tmp\") pod \"insights-operator-76bcd6ccf5-kk7mn\" (UID: \"21f9ed82-8d7f-48dc-af6f-a22d73178304\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn"
Sep 10 15:43:09.169111 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.169045    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21f9ed82-8d7f-48dc-af6f-a22d73178304-serving-cert\") pod \"insights-operator-76bcd6ccf5-kk7mn\" (UID: \"21f9ed82-8d7f-48dc-af6f-a22d73178304\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn"
Sep 10 15:43:09.169111 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.169077    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/811b2e5a-e4b6-45d9-99d4-4669a476afbc-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-nsnkp\" (UID: \"811b2e5a-e4b6-45d9-99d4-4669a476afbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-nsnkp"
Sep 10 15:43:09.169111 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.169102    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21f9ed82-8d7f-48dc-af6f-a22d73178304-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-kk7mn\" (UID: \"21f9ed82-8d7f-48dc-af6f-a22d73178304\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn"
Sep 10 15:43:09.169398 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.169131    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5c6t\" (UniqueName: \"kubernetes.io/projected/811b2e5a-e4b6-45d9-99d4-4669a476afbc-kube-api-access-z5c6t\") pod \"kube-storage-version-migrator-operator-8d7dd4675-nsnkp\" (UID: \"811b2e5a-e4b6-45d9-99d4-4669a476afbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-nsnkp"
Sep 10 15:43:09.169398 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.169194    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21f9ed82-8d7f-48dc-af6f-a22d73178304-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-kk7mn\" (UID: \"21f9ed82-8d7f-48dc-af6f-a22d73178304\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn"
Sep 10 15:43:09.169398 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.169215    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/21f9ed82-8d7f-48dc-af6f-a22d73178304-snapshots\") pod \"insights-operator-76bcd6ccf5-kk7mn\" (UID: \"21f9ed82-8d7f-48dc-af6f-a22d73178304\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn"
Sep 10 15:43:09.169545 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.169444    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21f9ed82-8d7f-48dc-af6f-a22d73178304-tmp\") pod \"insights-operator-76bcd6ccf5-kk7mn\" (UID: \"21f9ed82-8d7f-48dc-af6f-a22d73178304\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn"
Sep 10 15:43:09.169755 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.169664    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/21f9ed82-8d7f-48dc-af6f-a22d73178304-snapshots\") pod \"insights-operator-76bcd6ccf5-kk7mn\" (UID: \"21f9ed82-8d7f-48dc-af6f-a22d73178304\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn"
Sep 10 15:43:09.169755 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.169680    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21f9ed82-8d7f-48dc-af6f-a22d73178304-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-kk7mn\" (UID: \"21f9ed82-8d7f-48dc-af6f-a22d73178304\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn"
Sep 10 15:43:09.169755 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.169745    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/811b2e5a-e4b6-45d9-99d4-4669a476afbc-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-nsnkp\" (UID: \"811b2e5a-e4b6-45d9-99d4-4669a476afbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-nsnkp"
Sep 10 15:43:09.170553 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.170523    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21f9ed82-8d7f-48dc-af6f-a22d73178304-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-kk7mn\" (UID: \"21f9ed82-8d7f-48dc-af6f-a22d73178304\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn"
Sep 10 15:43:09.172477 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.172444    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/811b2e5a-e4b6-45d9-99d4-4669a476afbc-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-nsnkp\" (UID: \"811b2e5a-e4b6-45d9-99d4-4669a476afbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-nsnkp"
Sep 10 15:43:09.172624 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.172601    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21f9ed82-8d7f-48dc-af6f-a22d73178304-serving-cert\") pod \"insights-operator-76bcd6ccf5-kk7mn\" (UID: \"21f9ed82-8d7f-48dc-af6f-a22d73178304\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn"
Sep 10 15:43:09.177655 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.177627    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5c6t\" (UniqueName: \"kubernetes.io/projected/811b2e5a-e4b6-45d9-99d4-4669a476afbc-kube-api-access-z5c6t\") pod \"kube-storage-version-migrator-operator-8d7dd4675-nsnkp\" (UID: \"811b2e5a-e4b6-45d9-99d4-4669a476afbc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-nsnkp"
Sep 10 15:43:09.178201 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.178180    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w86n\" (UniqueName: \"kubernetes.io/projected/21f9ed82-8d7f-48dc-af6f-a22d73178304-kube-api-access-8w86n\") pod \"insights-operator-76bcd6ccf5-kk7mn\" (UID: \"21f9ed82-8d7f-48dc-af6f-a22d73178304\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn"
Sep 10 15:43:09.204782 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.204749    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-snm4z"
Sep 10 15:43:09.220823 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.220701    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-g45nr"
Sep 10 15:43:09.307818 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.307582    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-nsnkp"
Sep 10 15:43:09.313139 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.312963    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn"
Sep 10 15:43:09.472235 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.471617    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c3c7c51-bd5e-4508-8998-804694391874-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-g952v\" (UID: \"1c3c7c51-bd5e-4508-8998-804694391874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-g952v"
Sep 10 15:43:09.472235 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:09.471795    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 15:43:09.472235 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:09.471871    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c3c7c51-bd5e-4508-8998-804694391874-samples-operator-tls podName:1c3c7c51-bd5e-4508-8998-804694391874 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:10.471852061 +0000 UTC m=+69.276171313 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c3c7c51-bd5e-4508-8998-804694391874-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-g952v" (UID: "1c3c7c51-bd5e-4508-8998-804694391874") : secret "samples-operator-tls" not found
Sep 10 15:43:09.489096 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.489063    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-snm4z"]
Sep 10 15:43:09.493595 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:43:09.493534    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bf300c_3f8a_4e28_bbdc_21eedfd5a12f.slice/crio-36c860b0bba9429168c0c29ef70a8233efbe6a476e459c71690c5333042d8285 WatchSource:0}: Error finding container 36c860b0bba9429168c0c29ef70a8233efbe6a476e459c71690c5333042d8285: Status 404 returned error can't find the container with id 36c860b0bba9429168c0c29ef70a8233efbe6a476e459c71690c5333042d8285
Sep 10 15:43:09.515065 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.515038    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-g45nr"]
Sep 10 15:43:09.519430 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:43:09.519399    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14920648_4049_47e8_983a_8b9d7f41182b.slice/crio-cbefc264e4254ed710da938d2bc3ff9fe7b75e2f37bd70372fab6c09176711f6 WatchSource:0}: Error finding container cbefc264e4254ed710da938d2bc3ff9fe7b75e2f37bd70372fab6c09176711f6: Status 404 returned error can't find the container with id cbefc264e4254ed710da938d2bc3ff9fe7b75e2f37bd70372fab6c09176711f6
Sep 10 15:43:09.572903 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.572829    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fba202c6-588a-4976-b518-e1e95ae87c61-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6qs4r\" (UID: \"fba202c6-588a-4976-b518-e1e95ae87c61\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r"
Sep 10 15:43:09.573062 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:09.572961    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 15:43:09.573062 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:09.573029    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fba202c6-588a-4976-b518-e1e95ae87c61-cluster-monitoring-operator-tls podName:fba202c6-588a-4976-b518-e1e95ae87c61 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:10.573011556 +0000 UTC m=+69.377330800 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fba202c6-588a-4976-b518-e1e95ae87c61-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6qs4r" (UID: "fba202c6-588a-4976-b518-e1e95ae87c61") : secret "cluster-monitoring-operator-tls" not found
Sep 10 15:43:09.600258 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.600230    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-nsnkp"]
Sep 10 15:43:09.602968 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.602932    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-kk7mn"]
Sep 10 15:43:09.603183 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:43:09.603138    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811b2e5a_e4b6_45d9_99d4_4669a476afbc.slice/crio-2e114a120b91cad3c4d8c52888051da7af828de64cdb65f46a159e0db39bb0e0 WatchSource:0}: Error finding container 2e114a120b91cad3c4d8c52888051da7af828de64cdb65f46a159e0db39bb0e0: Status 404 returned error can't find the container with id 2e114a120b91cad3c4d8c52888051da7af828de64cdb65f46a159e0db39bb0e0
Sep 10 15:43:09.607320 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:43:09.607255    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f9ed82_8d7f_48dc_af6f_a22d73178304.slice/crio-932a77b72fd6d850f076b867bbf80ade2efb8d2a2ca4d28c4dca095ce75e5407 WatchSource:0}: Error finding container 932a77b72fd6d850f076b867bbf80ade2efb8d2a2ca4d28c4dca095ce75e5407: Status 404 returned error can't find the container with id 932a77b72fd6d850f076b867bbf80ade2efb8d2a2ca4d28c4dca095ce75e5407
Sep 10 15:43:09.903822 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.903723    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-nsnkp" event={"ID":"811b2e5a-e4b6-45d9-99d4-4669a476afbc","Type":"ContainerStarted","Data":"2e114a120b91cad3c4d8c52888051da7af828de64cdb65f46a159e0db39bb0e0"}
Sep 10 15:43:09.905384 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.905331    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn" event={"ID":"21f9ed82-8d7f-48dc-af6f-a22d73178304","Type":"ContainerStarted","Data":"932a77b72fd6d850f076b867bbf80ade2efb8d2a2ca4d28c4dca095ce75e5407"}
Sep 10 15:43:09.906705 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.906672    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-g45nr" event={"ID":"14920648-4049-47e8-983a-8b9d7f41182b","Type":"ContainerStarted","Data":"cbefc264e4254ed710da938d2bc3ff9fe7b75e2f37bd70372fab6c09176711f6"}
Sep 10 15:43:09.907977 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:09.907959    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-snm4z" event={"ID":"f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f","Type":"ContainerStarted","Data":"36c860b0bba9429168c0c29ef70a8233efbe6a476e459c71690c5333042d8285"}
Sep 10 15:43:10.480197 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:10.480108    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c3c7c51-bd5e-4508-8998-804694391874-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-g952v\" (UID: \"1c3c7c51-bd5e-4508-8998-804694391874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-g952v"
Sep 10 15:43:10.481401 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:10.480849    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 15:43:10.481401 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:10.480938    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c3c7c51-bd5e-4508-8998-804694391874-samples-operator-tls podName:1c3c7c51-bd5e-4508-8998-804694391874 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:12.48091611 +0000 UTC m=+71.285235371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c3c7c51-bd5e-4508-8998-804694391874-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-g952v" (UID: "1c3c7c51-bd5e-4508-8998-804694391874") : secret "samples-operator-tls" not found
Sep 10 15:43:10.582819 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:10.582785    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fba202c6-588a-4976-b518-e1e95ae87c61-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6qs4r\" (UID: \"fba202c6-588a-4976-b518-e1e95ae87c61\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r"
Sep 10 15:43:10.582989 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:10.582951    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 15:43:10.583065 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:10.583027    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fba202c6-588a-4976-b518-e1e95ae87c61-cluster-monitoring-operator-tls podName:fba202c6-588a-4976-b518-e1e95ae87c61 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:12.583005234 +0000 UTC m=+71.387324497 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fba202c6-588a-4976-b518-e1e95ae87c61-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6qs4r" (UID: "fba202c6-588a-4976-b518-e1e95ae87c61") : secret "cluster-monitoring-operator-tls" not found
Sep 10 15:43:12.498763 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:12.498720    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c3c7c51-bd5e-4508-8998-804694391874-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-g952v\" (UID: \"1c3c7c51-bd5e-4508-8998-804694391874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-g952v"
Sep 10 15:43:12.499225 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:12.498915    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 15:43:12.499225 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:12.499003    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c3c7c51-bd5e-4508-8998-804694391874-samples-operator-tls podName:1c3c7c51-bd5e-4508-8998-804694391874 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:16.498981485 +0000 UTC m=+75.303300729 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c3c7c51-bd5e-4508-8998-804694391874-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-g952v" (UID: "1c3c7c51-bd5e-4508-8998-804694391874") : secret "samples-operator-tls" not found
Sep 10 15:43:12.599482 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:12.599441    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fba202c6-588a-4976-b518-e1e95ae87c61-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6qs4r\" (UID: \"fba202c6-588a-4976-b518-e1e95ae87c61\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r"
Sep 10 15:43:12.599693 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:12.599625    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 15:43:12.599758 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:12.599723    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fba202c6-588a-4976-b518-e1e95ae87c61-cluster-monitoring-operator-tls podName:fba202c6-588a-4976-b518-e1e95ae87c61 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:16.599701217 +0000 UTC m=+75.404020475 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fba202c6-588a-4976-b518-e1e95ae87c61-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6qs4r" (UID: "fba202c6-588a-4976-b518-e1e95ae87c61") : secret "cluster-monitoring-operator-tls" not found
Sep 10 15:43:15.930751 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:15.930709    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g5l6v" event={"ID":"f5ced900-9635-4cf4-9390-7396c3ef86a7","Type":"ContainerStarted","Data":"c277d4eb68f79c0caa676114e316fa664b34c1794b0f2e2e6781dd3a82a3aa90"}
Sep 10 15:43:15.931247 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:15.930831    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:43:15.932501 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:15.932465    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-nsnkp" event={"ID":"811b2e5a-e4b6-45d9-99d4-4669a476afbc","Type":"ContainerStarted","Data":"3f007b1cd3b21f5a16356b7b7cbdd4fd44fec2218b5391289bb9bc011bb28155"}
Sep 10 15:43:15.934139 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:15.934103    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn" event={"ID":"21f9ed82-8d7f-48dc-af6f-a22d73178304","Type":"ContainerStarted","Data":"42fd9dc993506e93d41e9058c2c5b6a9804bd5c7b5e2771750ba07faf85c88f6"}
Sep 10 15:43:15.935766 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:15.935745    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-g45nr" event={"ID":"14920648-4049-47e8-983a-8b9d7f41182b","Type":"ContainerStarted","Data":"7a6e266dfbb3fe9941e77c46bcf2fca1d0bef9fd5526678ffae653b4a30f3b7e"}
Sep 10 15:43:15.937345 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:15.937325    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-snm4z_f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f/console-operator/0.log"
Sep 10 15:43:15.937430 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:15.937369    2564 generic.go:356] "Generic (PLEG): container finished" podID="f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f" containerID="520343c6a6e50fe1b8948772366ce324ac126803193e456a1a6a1a92e0cc11e6" exitCode=255
Sep 10 15:43:15.937476 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:15.937450    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-snm4z" event={"ID":"f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f","Type":"ContainerDied","Data":"520343c6a6e50fe1b8948772366ce324ac126803193e456a1a6a1a92e0cc11e6"}
Sep 10 15:43:15.937636 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:15.937620    2564 scope.go:117] "RemoveContainer" containerID="520343c6a6e50fe1b8948772366ce324ac126803193e456a1a6a1a92e0cc11e6"
Sep 10 15:43:15.939063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:15.939036    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-znbph" event={"ID":"8f29f7a6-d451-487d-92f1-84c47bf32d5c","Type":"ContainerStarted","Data":"1d795fe24f2af8fac252f8b06ac9b86ffad3f2a56f83302bb5d44a2eaa62c69a"}
Sep 10 15:43:15.946468 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:15.946421    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g5l6v" podStartSLOduration=66.837277626 podStartE2EDuration="1m13.946404953s" totalImagesPullingTime="7.109127327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:15.945610165 +0000 UTC m=+74.749929417" watchObservedRunningTime="2026-09-10 15:43:15.946404953 +0000 UTC m=+74.750724215"
Sep 10 15:43:15.993762 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:15.993717    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn" podStartSLOduration=2.324392383 podStartE2EDuration="7.993700112s" totalImagesPullingTime="5.669307729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:15.992691062 +0000 UTC m=+74.797010325" watchObservedRunningTime="2026-09-10 15:43:15.993700112 +0000 UTC m=+74.798019391"
Sep 10 15:43:16.011953 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:16.011318    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-g45nr" podStartSLOduration=2.260111495 podStartE2EDuration="8.01129802s" totalImagesPullingTime="5.751186525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:16.010656686 +0000 UTC m=+74.814975949" watchObservedRunningTime="2026-09-10 15:43:16.01129802 +0000 UTC m=+74.815617283"
Sep 10 15:43:16.045262 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:16.045204    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-znbph" podStartSLOduration=66.622666321 podStartE2EDuration="1m14.045182198s" totalImagesPullingTime="7.422515877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:16.028304198 +0000 UTC m=+74.832623460" watchObservedRunningTime="2026-09-10 15:43:16.045182198 +0000 UTC m=+74.849501461"
Sep 10 15:43:16.046173 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:16.046124    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-nsnkp" podStartSLOduration=2.378825718 podStartE2EDuration="8.046115532s" totalImagesPullingTime="5.667289814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:16.044886739 +0000 UTC m=+74.849206002" watchObservedRunningTime="2026-09-10 15:43:16.046115532 +0000 UTC m=+74.850434794"
Sep 10 15:43:16.534730 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:16.534691    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c3c7c51-bd5e-4508-8998-804694391874-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-g952v\" (UID: \"1c3c7c51-bd5e-4508-8998-804694391874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-g952v"
Sep 10 15:43:16.534922 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:16.534802    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 15:43:16.534922 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:16.534862    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c3c7c51-bd5e-4508-8998-804694391874-samples-operator-tls podName:1c3c7c51-bd5e-4508-8998-804694391874 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:24.534847665 +0000 UTC m=+83.339166917 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c3c7c51-bd5e-4508-8998-804694391874-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-g952v" (UID: "1c3c7c51-bd5e-4508-8998-804694391874") : secret "samples-operator-tls" not found
Sep 10 15:43:16.635623 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:16.635518    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fba202c6-588a-4976-b518-e1e95ae87c61-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6qs4r\" (UID: \"fba202c6-588a-4976-b518-e1e95ae87c61\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r"
Sep 10 15:43:16.635802 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:16.635697    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 15:43:16.635802 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:16.635798    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fba202c6-588a-4976-b518-e1e95ae87c61-cluster-monitoring-operator-tls podName:fba202c6-588a-4976-b518-e1e95ae87c61 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:24.635776463 +0000 UTC m=+83.440095715 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fba202c6-588a-4976-b518-e1e95ae87c61-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6qs4r" (UID: "fba202c6-588a-4976-b518-e1e95ae87c61") : secret "cluster-monitoring-operator-tls" not found
Sep 10 15:43:16.923868 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:16.923835    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pr6k5"]
Sep 10 15:43:16.925941 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:16.925920    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:16.928306 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:16.928275    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 15:43:16.928424 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:16.928306    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 15:43:16.928424 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:16.928342    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-89tcq\""
Sep 10 15:43:16.936055 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:16.936031    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pr6k5"]
Sep 10 15:43:16.944766 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:16.944746    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-snm4z_f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f/console-operator/1.log"
Sep 10 15:43:16.945245 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:16.945232    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-snm4z_f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f/console-operator/0.log"
Sep 10 15:43:16.945305 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:16.945264    2564 generic.go:356] "Generic (PLEG): container finished" podID="f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f" containerID="390d8aa46fa33d7aea317f3f73f420369b1036611d8127d4d20069eb4c762662" exitCode=255
Sep 10 15:43:16.945347 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:16.945336    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-snm4z" event={"ID":"f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f","Type":"ContainerDied","Data":"390d8aa46fa33d7aea317f3f73f420369b1036611d8127d4d20069eb4c762662"}
Sep 10 15:43:16.945407 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:16.945395    2564 scope.go:117] "RemoveContainer" containerID="520343c6a6e50fe1b8948772366ce324ac126803193e456a1a6a1a92e0cc11e6"
Sep 10 15:43:16.945565 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:16.945551    2564 scope.go:117] "RemoveContainer" containerID="390d8aa46fa33d7aea317f3f73f420369b1036611d8127d4d20069eb4c762662"
Sep 10 15:43:16.945749 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:16.945730    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-snm4z_openshift-console-operator(f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f)\"" pod="openshift-console-operator/console-operator-85b454874b-snm4z" podUID="f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f"
Sep 10 15:43:17.038688 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:17.038650    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7d601040-aa12-4aab-85df-a2c78ddc3f68-crio-socket\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:17.038852 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:17.038772    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d601040-aa12-4aab-85df-a2c78ddc3f68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:17.038852 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:17.038811    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh6ts\" (UniqueName: \"kubernetes.io/projected/7d601040-aa12-4aab-85df-a2c78ddc3f68-kube-api-access-zh6ts\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:17.039203 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:17.039147    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d601040-aa12-4aab-85df-a2c78ddc3f68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:17.039353 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:17.039296    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d601040-aa12-4aab-85df-a2c78ddc3f68-data-volume\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:17.139741 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:17.139704    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7d601040-aa12-4aab-85df-a2c78ddc3f68-crio-socket\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:17.139919 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:17.139763    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d601040-aa12-4aab-85df-a2c78ddc3f68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:17.139919 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:17.139788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh6ts\" (UniqueName: \"kubernetes.io/projected/7d601040-aa12-4aab-85df-a2c78ddc3f68-kube-api-access-zh6ts\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:17.139919 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:17.139833    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7d601040-aa12-4aab-85df-a2c78ddc3f68-crio-socket\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:17.139919 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:17.139851    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d601040-aa12-4aab-85df-a2c78ddc3f68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:17.139919 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:17.139899    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d601040-aa12-4aab-85df-a2c78ddc3f68-data-volume\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:17.140197 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:17.139925    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:43:17.140197 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:17.139969    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:43:17.140197 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:17.139992    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d601040-aa12-4aab-85df-a2c78ddc3f68-insights-runtime-extractor-tls podName:7d601040-aa12-4aab-85df-a2c78ddc3f68 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:17.63997613 +0000 UTC m=+76.444295368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7d601040-aa12-4aab-85df-a2c78ddc3f68-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pr6k5" (UID: "7d601040-aa12-4aab-85df-a2c78ddc3f68") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:43:17.140197 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:17.140011    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d601040-aa12-4aab-85df-a2c78ddc3f68-kube-rbac-proxy-cm podName:7d601040-aa12-4aab-85df-a2c78ddc3f68 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:17.63999948 +0000 UTC m=+76.444318719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7d601040-aa12-4aab-85df-a2c78ddc3f68-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pr6k5" (UID: "7d601040-aa12-4aab-85df-a2c78ddc3f68") : configmap "kube-rbac-proxy" not found
Sep 10 15:43:17.140383 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:17.140291    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d601040-aa12-4aab-85df-a2c78ddc3f68-data-volume\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:17.149424 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:17.149404    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh6ts\" (UniqueName: \"kubernetes.io/projected/7d601040-aa12-4aab-85df-a2c78ddc3f68-kube-api-access-zh6ts\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:17.646857 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:17.643995    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d601040-aa12-4aab-85df-a2c78ddc3f68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:17.646857 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:17.644147    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d601040-aa12-4aab-85df-a2c78ddc3f68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:17.646857 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:17.644373    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:43:17.646857 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:17.644460    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:43:17.646857 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:17.644478    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d601040-aa12-4aab-85df-a2c78ddc3f68-insights-runtime-extractor-tls podName:7d601040-aa12-4aab-85df-a2c78ddc3f68 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:18.64445663 +0000 UTC m=+77.448775889 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7d601040-aa12-4aab-85df-a2c78ddc3f68-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pr6k5" (UID: "7d601040-aa12-4aab-85df-a2c78ddc3f68") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:43:17.646857 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:17.644507    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d601040-aa12-4aab-85df-a2c78ddc3f68-kube-rbac-proxy-cm podName:7d601040-aa12-4aab-85df-a2c78ddc3f68 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:18.644487937 +0000 UTC m=+77.448807195 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7d601040-aa12-4aab-85df-a2c78ddc3f68-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pr6k5" (UID: "7d601040-aa12-4aab-85df-a2c78ddc3f68") : configmap "kube-rbac-proxy" not found
Sep 10 15:43:17.950053 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:17.950024    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-snm4z_f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f/console-operator/1.log"
Sep 10 15:43:17.950468 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:17.950358    2564 scope.go:117] "RemoveContainer" containerID="390d8aa46fa33d7aea317f3f73f420369b1036611d8127d4d20069eb4c762662"
Sep 10 15:43:17.950532 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:17.950512    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-snm4z_openshift-console-operator(f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f)\"" pod="openshift-console-operator/console-operator-85b454874b-snm4z" podUID="f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f"
Sep 10 15:43:18.651761 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:18.651719    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d601040-aa12-4aab-85df-a2c78ddc3f68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:18.651947 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:18.651782    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d601040-aa12-4aab-85df-a2c78ddc3f68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:18.651947 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:18.651856    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:43:18.651947 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:18.651881    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:43:18.651947 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:18.651913    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d601040-aa12-4aab-85df-a2c78ddc3f68-kube-rbac-proxy-cm podName:7d601040-aa12-4aab-85df-a2c78ddc3f68 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:20.651899163 +0000 UTC m=+79.456218401 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7d601040-aa12-4aab-85df-a2c78ddc3f68-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pr6k5" (UID: "7d601040-aa12-4aab-85df-a2c78ddc3f68") : configmap "kube-rbac-proxy" not found
Sep 10 15:43:18.651947 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:18.651936    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d601040-aa12-4aab-85df-a2c78ddc3f68-insights-runtime-extractor-tls podName:7d601040-aa12-4aab-85df-a2c78ddc3f68 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:20.651923017 +0000 UTC m=+79.456242256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7d601040-aa12-4aab-85df-a2c78ddc3f68-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pr6k5" (UID: "7d601040-aa12-4aab-85df-a2c78ddc3f68") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:43:19.205178 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:19.205123    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-snm4z"
Sep 10 15:43:19.205532 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:19.205205    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-snm4z"
Sep 10 15:43:19.205532 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:19.205509    2564 scope.go:117] "RemoveContainer" containerID="390d8aa46fa33d7aea317f3f73f420369b1036611d8127d4d20069eb4c762662"
Sep 10 15:43:19.205683 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:19.205663    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-snm4z_openshift-console-operator(f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f)\"" pod="openshift-console-operator/console-operator-85b454874b-snm4z" podUID="f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f"
Sep 10 15:43:19.884542 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:19.884512    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hb6s4_c5fe8c6d-b8ce-4138-b5ed-a05499ccd346/dns-node-resolver/0.log"
Sep 10 15:43:20.668170 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:20.668131    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d601040-aa12-4aab-85df-a2c78ddc3f68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:20.668582 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:20.668218    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d601040-aa12-4aab-85df-a2c78ddc3f68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:20.668582 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:20.668291    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:43:20.668582 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:20.668294    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:43:20.668582 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:20.668344    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d601040-aa12-4aab-85df-a2c78ddc3f68-kube-rbac-proxy-cm podName:7d601040-aa12-4aab-85df-a2c78ddc3f68 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:24.668331145 +0000 UTC m=+83.472650384 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7d601040-aa12-4aab-85df-a2c78ddc3f68-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pr6k5" (UID: "7d601040-aa12-4aab-85df-a2c78ddc3f68") : configmap "kube-rbac-proxy" not found
Sep 10 15:43:20.668582 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:20.668357    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d601040-aa12-4aab-85df-a2c78ddc3f68-insights-runtime-extractor-tls podName:7d601040-aa12-4aab-85df-a2c78ddc3f68 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:24.668351653 +0000 UTC m=+83.472670892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7d601040-aa12-4aab-85df-a2c78ddc3f68-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pr6k5" (UID: "7d601040-aa12-4aab-85df-a2c78ddc3f68") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:43:20.884253 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:20.884225    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k6fhq_e6f2f68d-170c-4567-ba06-b05bea2cb642/node-ca/0.log"
Sep 10 15:43:21.285675 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:21.285638    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8bc5c_28343b32-70a8-4d77-8dfd-d7b27a3f8cef/serve-healthcheck-canary/0.log"
Sep 10 15:43:22.286717 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:22.286685    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-nsnkp_811b2e5a-e4b6-45d9-99d4-4669a476afbc/kube-storage-version-migrator-operator/0.log"
Sep 10 15:43:24.597648 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:24.597597    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c3c7c51-bd5e-4508-8998-804694391874-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-g952v\" (UID: \"1c3c7c51-bd5e-4508-8998-804694391874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-g952v"
Sep 10 15:43:24.600087 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:24.600060    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c3c7c51-bd5e-4508-8998-804694391874-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-g952v\" (UID: \"1c3c7c51-bd5e-4508-8998-804694391874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-g952v"
Sep 10 15:43:24.698578 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:24.698546    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d601040-aa12-4aab-85df-a2c78ddc3f68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:24.698578 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:24.698587    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fba202c6-588a-4976-b518-e1e95ae87c61-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6qs4r\" (UID: \"fba202c6-588a-4976-b518-e1e95ae87c61\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r"
Sep 10 15:43:24.698779 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:24.698618    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d601040-aa12-4aab-85df-a2c78ddc3f68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:24.698779 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:24.698689    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:43:24.698779 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:24.698698    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:43:24.698779 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:24.698703    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 15:43:24.698779 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:24.698753    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d601040-aa12-4aab-85df-a2c78ddc3f68-kube-rbac-proxy-cm podName:7d601040-aa12-4aab-85df-a2c78ddc3f68 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:32.698738025 +0000 UTC m=+91.503057263 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7d601040-aa12-4aab-85df-a2c78ddc3f68-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pr6k5" (UID: "7d601040-aa12-4aab-85df-a2c78ddc3f68") : configmap "kube-rbac-proxy" not found
Sep 10 15:43:24.698779 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:24.698765    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d601040-aa12-4aab-85df-a2c78ddc3f68-insights-runtime-extractor-tls podName:7d601040-aa12-4aab-85df-a2c78ddc3f68 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:32.698759425 +0000 UTC m=+91.503078664 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7d601040-aa12-4aab-85df-a2c78ddc3f68-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pr6k5" (UID: "7d601040-aa12-4aab-85df-a2c78ddc3f68") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:43:24.698779 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:24.698776    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fba202c6-588a-4976-b518-e1e95ae87c61-cluster-monitoring-operator-tls podName:fba202c6-588a-4976-b518-e1e95ae87c61 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:40.698769812 +0000 UTC m=+99.503089050 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fba202c6-588a-4976-b518-e1e95ae87c61-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-6qs4r" (UID: "fba202c6-588a-4976-b518-e1e95ae87c61") : secret "cluster-monitoring-operator-tls" not found
Sep 10 15:43:24.703118 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:24.703100    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-g952v"
Sep 10 15:43:24.973501 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:24.973285    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-g952v"]
Sep 10 15:43:25.975433 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:25.975396    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-g952v" event={"ID":"1c3c7c51-bd5e-4508-8998-804694391874","Type":"ContainerStarted","Data":"42797127c4fb4a92e692f51da82e3e306879f4c682fd436a5c7426e0dba47a4a"}
Sep 10 15:43:26.980445 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:26.980405    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-g952v" event={"ID":"1c3c7c51-bd5e-4508-8998-804694391874","Type":"ContainerStarted","Data":"76ec97bee835ffdcf7e977ab58fd31a00159d82c6495908c38b2e3e3de7c7159"}
Sep 10 15:43:26.980813 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:26.980495    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-g952v" event={"ID":"1c3c7c51-bd5e-4508-8998-804694391874","Type":"ContainerStarted","Data":"b69c5197e2204bca1adf1cfc28439311cbfb00b24668406b6fab967c07a6a0b6"}
Sep 10 15:43:26.997534 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:26.997487    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-g952v" podStartSLOduration=17.370964048 podStartE2EDuration="18.997471933s" totalImagesPullingTime="1.626507885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:26.997331121 +0000 UTC m=+85.801650372" watchObservedRunningTime="2026-09-10 15:43:26.997471933 +0000 UTC m=+85.801791193"
Sep 10 15:43:29.666693 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:29.666659    2564 scope.go:117] "RemoveContainer" containerID="390d8aa46fa33d7aea317f3f73f420369b1036611d8127d4d20069eb4c762662"
Sep 10 15:43:29.991230 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:29.991205    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-snm4z_f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f/console-operator/1.log"
Sep 10 15:43:29.991387 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:29.991294    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-snm4z" event={"ID":"f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f","Type":"ContainerStarted","Data":"60471d5bc67f2a4ac11cee912d8c9878367e7203484566b5ddb63ba73060f719"}
Sep 10 15:43:29.991590 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:29.991560    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-snm4z"
Sep 10 15:43:30.010111 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:30.010060    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-snm4z" podStartSLOduration=16.231300724 podStartE2EDuration="22.010043269s" totalImagesPullingTime="5.778742545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:30.009717692 +0000 UTC m=+88.814037141" watchObservedRunningTime="2026-09-10 15:43:30.010043269 +0000 UTC m=+88.814362530"
Sep 10 15:43:30.015246 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:30.015222    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-snm4z"
Sep 10 15:43:32.767771 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:32.767730    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d601040-aa12-4aab-85df-a2c78ddc3f68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:32.768196 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:32.767809    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d601040-aa12-4aab-85df-a2c78ddc3f68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:32.768378 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:32.768356    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7d601040-aa12-4aab-85df-a2c78ddc3f68-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:32.770203 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:32.770185    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7d601040-aa12-4aab-85df-a2c78ddc3f68-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pr6k5\" (UID: \"7d601040-aa12-4aab-85df-a2c78ddc3f68\") " pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:32.835084 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:32.835052    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pr6k5"
Sep 10 15:43:33.121229 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:33.121198    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pr6k5"]
Sep 10 15:43:34.005271 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:34.005230    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pr6k5" event={"ID":"7d601040-aa12-4aab-85df-a2c78ddc3f68","Type":"ContainerStarted","Data":"3e4576655941b321921e6299c9d8176e943fee7cf0b952e2a8974601c1f454bd"}
Sep 10 15:43:34.005619 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:34.005299    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pr6k5" event={"ID":"7d601040-aa12-4aab-85df-a2c78ddc3f68","Type":"ContainerStarted","Data":"05ea8febb5e432f38d4d3257ce173fa18de3bca58263e681413c9417b71f2e41"}
Sep 10 15:43:34.005619 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:34.005324    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pr6k5" event={"ID":"7d601040-aa12-4aab-85df-a2c78ddc3f68","Type":"ContainerStarted","Data":"b8b73feae3ea64f2a74fc0470a5747d8d60f11e5e7fec9e071fd3aa607dc30e8"}
Sep 10 15:43:36.015085 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:36.015050    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pr6k5" event={"ID":"7d601040-aa12-4aab-85df-a2c78ddc3f68","Type":"ContainerStarted","Data":"03834e6dd9379d25fa3376dc22d76d21095da7a4f64ff899a7d19043ecf1ded7"}
Sep 10 15:43:36.032704 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:36.032642    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pr6k5" podStartSLOduration=17.978855629 podStartE2EDuration="20.032626471s" totalImagesPullingTime="2.053770842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:43:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:36.031231061 +0000 UTC m=+94.835550324" watchObservedRunningTime="2026-09-10 15:43:36.032626471 +0000 UTC m=+94.836945732"
Sep 10 15:43:40.523551 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:40.523505    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls\") pod \"dns-default-2qrbr\" (UID: \"db9487f8-8adc-4d2e-8c5b-7222bf915898\") " pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:43:40.526050 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:40.526024    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db9487f8-8adc-4d2e-8c5b-7222bf915898-metrics-tls\") pod \"dns-default-2qrbr\" (UID: \"db9487f8-8adc-4d2e-8c5b-7222bf915898\") " pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:43:40.724821 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:40.724775    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fba202c6-588a-4976-b518-e1e95ae87c61-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6qs4r\" (UID: \"fba202c6-588a-4976-b518-e1e95ae87c61\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r"
Sep 10 15:43:40.727310 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:40.727285    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fba202c6-588a-4976-b518-e1e95ae87c61-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-6qs4r\" (UID: \"fba202c6-588a-4976-b518-e1e95ae87c61\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r"
Sep 10 15:43:40.804496 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:40.804404    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z4knc\""
Sep 10 15:43:40.812438 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:40.812413    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:43:41.013359 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:41.013322    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r"
Sep 10 15:43:41.091048 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:41.090994    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2qrbr"]
Sep 10 15:43:41.093600 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:43:41.093578    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb9487f8_8adc_4d2e_8c5b_7222bf915898.slice/crio-5652450f53cc37da6f1c791bb16c20915e11719117833f5ddc737fd56b5ce744 WatchSource:0}: Error finding container 5652450f53cc37da6f1c791bb16c20915e11719117833f5ddc737fd56b5ce744: Status 404 returned error can't find the container with id 5652450f53cc37da6f1c791bb16c20915e11719117833f5ddc737fd56b5ce744
Sep 10 15:43:41.285530 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:41.285497    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r"]
Sep 10 15:43:41.288407 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:43:41.288377    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba202c6_588a_4976_b518_e1e95ae87c61.slice/crio-388b9bf3ed57fbf8a5cdf2e103c68fbce95df25848b879dc6448ffe0b34e5201 WatchSource:0}: Error finding container 388b9bf3ed57fbf8a5cdf2e103c68fbce95df25848b879dc6448ffe0b34e5201: Status 404 returned error can't find the container with id 388b9bf3ed57fbf8a5cdf2e103c68fbce95df25848b879dc6448ffe0b34e5201
Sep 10 15:43:42.037262 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.037219    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r" event={"ID":"fba202c6-588a-4976-b518-e1e95ae87c61","Type":"ContainerStarted","Data":"388b9bf3ed57fbf8a5cdf2e103c68fbce95df25848b879dc6448ffe0b34e5201"}
Sep 10 15:43:42.038838 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.038780    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2qrbr" event={"ID":"db9487f8-8adc-4d2e-8c5b-7222bf915898","Type":"ContainerStarted","Data":"5652450f53cc37da6f1c791bb16c20915e11719117833f5ddc737fd56b5ce744"}
Sep 10 15:43:42.367804 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.367719    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-8fh95"]
Sep 10 15:43:42.372127 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.372100    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-8fh95"
Sep 10 15:43:42.374446 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.374420    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-tztt4\""
Sep 10 15:43:42.379733 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.379211    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-8fh95"]
Sep 10 15:43:42.438342 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.438311    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c56w8\" (UniqueName: \"kubernetes.io/projected/82ee38da-6534-4eee-a598-0c167d07cf6c-kube-api-access-c56w8\") pod \"network-check-source-685456655f-8fh95\" (UID: \"82ee38da-6534-4eee-a598-0c167d07cf6c\") " pod="openshift-network-diagnostics/network-check-source-685456655f-8fh95"
Sep 10 15:43:42.490188 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.489191    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-qlzmm"]
Sep 10 15:43:42.494845 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.494817    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-qlzmm"
Sep 10 15:43:42.497905 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.497527    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 15:43:42.497905 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.497642    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 15:43:42.497905 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.497761    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-55bf9\""
Sep 10 15:43:42.503026 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.502961    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-qlzmm"]
Sep 10 15:43:42.539805 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.539725    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c56w8\" (UniqueName: \"kubernetes.io/projected/82ee38da-6534-4eee-a598-0c167d07cf6c-kube-api-access-c56w8\") pod \"network-check-source-685456655f-8fh95\" (UID: \"82ee38da-6534-4eee-a598-0c167d07cf6c\") " pod="openshift-network-diagnostics/network-check-source-685456655f-8fh95"
Sep 10 15:43:42.550744 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.550683    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c56w8\" (UniqueName: \"kubernetes.io/projected/82ee38da-6534-4eee-a598-0c167d07cf6c-kube-api-access-c56w8\") pod \"network-check-source-685456655f-8fh95\" (UID: \"82ee38da-6534-4eee-a598-0c167d07cf6c\") " pod="openshift-network-diagnostics/network-check-source-685456655f-8fh95"
Sep 10 15:43:42.565531 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.565498    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55549cd8fb-mrpz8"]
Sep 10 15:43:42.568969 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.568947    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:42.571621 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.571595    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lmmsg\""
Sep 10 15:43:42.571736 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.571627    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 15:43:42.571736 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.571657    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 15:43:42.571818 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.571804    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 15:43:42.571902 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.571878    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 15:43:42.572008 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.571987    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 15:43:42.580615 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.579975    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55549cd8fb-mrpz8"]
Sep 10 15:43:42.584835 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.584552    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cf899df5c-trxv8"]
Sep 10 15:43:42.587911 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.587891    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.593487 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.593190    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 15:43:42.593487 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.593247    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 15:43:42.593487 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.593289    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-whst8\""
Sep 10 15:43:42.593879 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.593863    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 15:43:42.612655 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.612627    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cf899df5c-trxv8"]
Sep 10 15:43:42.620626 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.620360    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 15:43:42.640269 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.640235    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-console-config\") pod \"console-55549cd8fb-mrpz8\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") " pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:42.640456 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.640287    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eae7786c-775a-4130-b011-415e5c1e18b8-registry-tls\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.640456 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.640316    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-console-oauth-config\") pod \"console-55549cd8fb-mrpz8\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") " pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:42.640456 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.640351    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eae7786c-775a-4130-b011-415e5c1e18b8-registry-certificates\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.640456 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.640377    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6jcw\" (UniqueName: \"kubernetes.io/projected/eae7786c-775a-4130-b011-415e5c1e18b8-kube-api-access-x6jcw\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.640456 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.640418    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eae7786c-775a-4130-b011-415e5c1e18b8-bound-sa-token\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.640456 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.640453    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eae7786c-775a-4130-b011-415e5c1e18b8-installation-pull-secrets\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.640742 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.640479    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-console-serving-cert\") pod \"console-55549cd8fb-mrpz8\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") " pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:42.640742 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.640503    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkx7t\" (UniqueName: \"kubernetes.io/projected/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-kube-api-access-bkx7t\") pod \"console-55549cd8fb-mrpz8\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") " pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:42.640742 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.640575    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-oauth-serving-cert\") pod \"console-55549cd8fb-mrpz8\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") " pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:42.640742 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.640609    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-service-ca\") pod \"console-55549cd8fb-mrpz8\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") " pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:42.640742 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.640658    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eae7786c-775a-4130-b011-415e5c1e18b8-image-registry-private-configuration\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.640742 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.640694    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfql8\" (UniqueName: \"kubernetes.io/projected/b8a041bb-e51d-4b49-92c8-e25b58a05f3a-kube-api-access-xfql8\") pod \"downloads-85d98c998f-qlzmm\" (UID: \"b8a041bb-e51d-4b49-92c8-e25b58a05f3a\") " pod="openshift-console/downloads-85d98c998f-qlzmm"
Sep 10 15:43:42.640742 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.640723    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eae7786c-775a-4130-b011-415e5c1e18b8-ca-trust-extracted\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.641046 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.640750    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eae7786c-775a-4130-b011-415e5c1e18b8-trusted-ca\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.684674 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.684636    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-8fh95"
Sep 10 15:43:42.742513 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.742181    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eae7786c-775a-4130-b011-415e5c1e18b8-registry-tls\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.742513 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.742218    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-console-oauth-config\") pod \"console-55549cd8fb-mrpz8\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") " pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:42.742513 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.742251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eae7786c-775a-4130-b011-415e5c1e18b8-registry-certificates\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.742513 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.742276    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6jcw\" (UniqueName: \"kubernetes.io/projected/eae7786c-775a-4130-b011-415e5c1e18b8-kube-api-access-x6jcw\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.742513 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.742306    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eae7786c-775a-4130-b011-415e5c1e18b8-bound-sa-token\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.742513 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.742333    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eae7786c-775a-4130-b011-415e5c1e18b8-installation-pull-secrets\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.742513 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.742355    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-console-serving-cert\") pod \"console-55549cd8fb-mrpz8\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") " pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:42.742513 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.742376    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkx7t\" (UniqueName: \"kubernetes.io/projected/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-kube-api-access-bkx7t\") pod \"console-55549cd8fb-mrpz8\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") " pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:42.742513 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.742401    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-oauth-serving-cert\") pod \"console-55549cd8fb-mrpz8\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") " pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:42.742513 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.742422    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-service-ca\") pod \"console-55549cd8fb-mrpz8\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") " pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:42.742513 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.742451    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eae7786c-775a-4130-b011-415e5c1e18b8-image-registry-private-configuration\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.742513 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.742487    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfql8\" (UniqueName: \"kubernetes.io/projected/b8a041bb-e51d-4b49-92c8-e25b58a05f3a-kube-api-access-xfql8\") pod \"downloads-85d98c998f-qlzmm\" (UID: \"b8a041bb-e51d-4b49-92c8-e25b58a05f3a\") " pod="openshift-console/downloads-85d98c998f-qlzmm"
Sep 10 15:43:42.742513 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.742511    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eae7786c-775a-4130-b011-415e5c1e18b8-ca-trust-extracted\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.743092 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.742530    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eae7786c-775a-4130-b011-415e5c1e18b8-trusted-ca\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.743092 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.742577    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-console-config\") pod \"console-55549cd8fb-mrpz8\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") " pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:42.743394 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.743367    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-console-config\") pod \"console-55549cd8fb-mrpz8\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") " pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:42.743897 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.743563    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-oauth-serving-cert\") pod \"console-55549cd8fb-mrpz8\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") " pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:42.743897 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.743639    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eae7786c-775a-4130-b011-415e5c1e18b8-registry-certificates\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.744056 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.743937    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-service-ca\") pod \"console-55549cd8fb-mrpz8\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") " pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:42.744269 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.744249    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eae7786c-775a-4130-b011-415e5c1e18b8-ca-trust-extracted\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.744827 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.744797    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eae7786c-775a-4130-b011-415e5c1e18b8-trusted-ca\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.748557 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.747456    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eae7786c-775a-4130-b011-415e5c1e18b8-image-registry-private-configuration\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.750019 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.749995    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eae7786c-775a-4130-b011-415e5c1e18b8-installation-pull-secrets\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.750526 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.750505    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eae7786c-775a-4130-b011-415e5c1e18b8-registry-tls\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.750925 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.750907    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-console-oauth-config\") pod \"console-55549cd8fb-mrpz8\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") " pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:42.753240 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.753194    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-console-serving-cert\") pod \"console-55549cd8fb-mrpz8\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") " pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:42.758725 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.758566    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eae7786c-775a-4130-b011-415e5c1e18b8-bound-sa-token\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.758725 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.758632    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfql8\" (UniqueName: \"kubernetes.io/projected/b8a041bb-e51d-4b49-92c8-e25b58a05f3a-kube-api-access-xfql8\") pod \"downloads-85d98c998f-qlzmm\" (UID: \"b8a041bb-e51d-4b49-92c8-e25b58a05f3a\") " pod="openshift-console/downloads-85d98c998f-qlzmm"
Sep 10 15:43:42.758725 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.758684    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkx7t\" (UniqueName: \"kubernetes.io/projected/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-kube-api-access-bkx7t\") pod \"console-55549cd8fb-mrpz8\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") " pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:42.760412 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.760391    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6jcw\" (UniqueName: \"kubernetes.io/projected/eae7786c-775a-4130-b011-415e5c1e18b8-kube-api-access-x6jcw\") pod \"image-registry-5cf899df5c-trxv8\" (UID: \"eae7786c-775a-4130-b011-415e5c1e18b8\") " pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.809811 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.809770    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-qlzmm"
Sep 10 15:43:42.883607 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.883515    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:42.900847 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.900810    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:42.958639 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:42.958604    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-8fh95"]
Sep 10 15:43:43.217785 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:43:43.217750    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ee38da_6534_4eee_a598_0c167d07cf6c.slice/crio-e067724d767429b1e15d836acf260ba5b25666552ade972f060a9c7b18ff79b9 WatchSource:0}: Error finding container e067724d767429b1e15d836acf260ba5b25666552ade972f060a9c7b18ff79b9: Status 404 returned error can't find the container with id e067724d767429b1e15d836acf260ba5b25666552ade972f060a9c7b18ff79b9
Sep 10 15:43:43.509943 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:43.509905    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cf899df5c-trxv8"]
Sep 10 15:43:43.513249 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:43:43.513211    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae7786c_775a_4130_b011_415e5c1e18b8.slice/crio-8749f4be53321d9f6daa599bcbee77c19fa12fd0a4c38a37b69dc2031097ddbb WatchSource:0}: Error finding container 8749f4be53321d9f6daa599bcbee77c19fa12fd0a4c38a37b69dc2031097ddbb: Status 404 returned error can't find the container with id 8749f4be53321d9f6daa599bcbee77c19fa12fd0a4c38a37b69dc2031097ddbb
Sep 10 15:43:43.530586 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:43.530562    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55549cd8fb-mrpz8"]
Sep 10 15:43:43.533637 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:43.533607    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-qlzmm"]
Sep 10 15:43:43.838404 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:43.838323    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-swfht"]
Sep 10 15:43:43.841899 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:43.841876    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-swfht"
Sep 10 15:43:43.843981 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:43.843959    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 10 15:43:43.844316 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:43.844297    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-tvq95\""
Sep 10 15:43:43.850667 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:43.850629    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-swfht"]
Sep 10 15:43:43.953858 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:43.953822    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8b39dc44-71c0-40cf-8837-dbb9a86be186-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-swfht\" (UID: \"8b39dc44-71c0-40cf-8837-dbb9a86be186\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-swfht"
Sep 10 15:43:44.049535 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:44.049466    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r" event={"ID":"fba202c6-588a-4976-b518-e1e95ae87c61","Type":"ContainerStarted","Data":"87dbf2e9391dafcc6548c952480f09f8a81f7186e054dab9d62b9b0262b6f03f"}
Sep 10 15:43:44.051984 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:44.051953    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2qrbr" event={"ID":"db9487f8-8adc-4d2e-8c5b-7222bf915898","Type":"ContainerStarted","Data":"d2b65f88c538814974cc1e1c8cae3fa1d94e6e9c6cc16e08c24c6de049eb8e0d"}
Sep 10 15:43:44.052150 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:44.052017    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2qrbr" event={"ID":"db9487f8-8adc-4d2e-8c5b-7222bf915898","Type":"ContainerStarted","Data":"5d6d5d087d445de15ee43776c109e9e4929341a4a3910932e809f76a7381b6ea"}
Sep 10 15:43:44.052150 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:44.052133    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:43:44.054086 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:44.054062    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-qlzmm" event={"ID":"b8a041bb-e51d-4b49-92c8-e25b58a05f3a","Type":"ContainerStarted","Data":"b6da9b748e9c4d2ce5db5cfb0c664b91872844a795beb0cb82a410da1729cc07"}
Sep 10 15:43:44.054342 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:44.054308    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8b39dc44-71c0-40cf-8837-dbb9a86be186-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-swfht\" (UID: \"8b39dc44-71c0-40cf-8837-dbb9a86be186\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-swfht"
Sep 10 15:43:44.054477 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:44.054414    2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 10 15:43:44.054554 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:44.054479    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b39dc44-71c0-40cf-8837-dbb9a86be186-tls-certificates podName:8b39dc44-71c0-40cf-8837-dbb9a86be186 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:44.554460831 +0000 UTC m=+103.358780085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/8b39dc44-71c0-40cf-8837-dbb9a86be186-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-swfht" (UID: "8b39dc44-71c0-40cf-8837-dbb9a86be186") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 10 15:43:44.057893 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:44.057869    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cf899df5c-trxv8" event={"ID":"eae7786c-775a-4130-b011-415e5c1e18b8","Type":"ContainerStarted","Data":"804b74f928f42e5e609358358a630314be876348d51dc99f966cfff5d0fb4cc8"}
Sep 10 15:43:44.057979 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:44.057909    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cf899df5c-trxv8" event={"ID":"eae7786c-775a-4130-b011-415e5c1e18b8","Type":"ContainerStarted","Data":"8749f4be53321d9f6daa599bcbee77c19fa12fd0a4c38a37b69dc2031097ddbb"}
Sep 10 15:43:44.057979 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:44.057949    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:43:44.059324 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:44.059302    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55549cd8fb-mrpz8" event={"ID":"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899","Type":"ContainerStarted","Data":"003c9a2ebcb47767fa023e97fc5dce6783b1957ab09ecf49f1ffc40e7696d32d"}
Sep 10 15:43:44.061389 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:44.061362    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-8fh95" event={"ID":"82ee38da-6534-4eee-a598-0c167d07cf6c","Type":"ContainerStarted","Data":"12be60f8c74daf649619530fc14ec9c10a1bd4437d790c72b685fb0fb571a014"}
Sep 10 15:43:44.061490 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:44.061402    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-8fh95" event={"ID":"82ee38da-6534-4eee-a598-0c167d07cf6c","Type":"ContainerStarted","Data":"e067724d767429b1e15d836acf260ba5b25666552ade972f060a9c7b18ff79b9"}
Sep 10 15:43:44.067640 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:44.067550    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-6qs4r" podStartSLOduration=34.081384377 podStartE2EDuration="36.067537061s" totalImagesPullingTime="1.986152684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:44.067070132 +0000 UTC m=+102.871389394" watchObservedRunningTime="2026-09-10 15:43:44.067537061 +0000 UTC m=+102.871856322"
Sep 10 15:43:44.106513 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:44.106140    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2qrbr" podStartSLOduration=65.971989463 podStartE2EDuration="1m8.106117904s" totalImagesPullingTime="2.134128441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:44.087515045 +0000 UTC m=+102.891834307" watchObservedRunningTime="2026-09-10 15:43:44.106117904 +0000 UTC m=+102.910437163"
Sep 10 15:43:44.106513 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:44.106330    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-8fh95" podStartSLOduration=2.106320109 podStartE2EDuration="2.106320109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:44.104261828 +0000 UTC m=+102.908581092" watchObservedRunningTime="2026-09-10 15:43:44.106320109 +0000 UTC m=+102.910639371"
Sep 10 15:43:44.124006 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:44.123926    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5cf899df5c-trxv8" podStartSLOduration=2.123906211 podStartE2EDuration="2.123906211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:44.122708462 +0000 UTC m=+102.927027725" watchObservedRunningTime="2026-09-10 15:43:44.123906211 +0000 UTC m=+102.928225473"
Sep 10 15:43:44.559719 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:44.559656    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8b39dc44-71c0-40cf-8837-dbb9a86be186-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-swfht\" (UID: \"8b39dc44-71c0-40cf-8837-dbb9a86be186\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-swfht"
Sep 10 15:43:44.560739 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:44.559813    2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 10 15:43:44.560739 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:44.559884    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b39dc44-71c0-40cf-8837-dbb9a86be186-tls-certificates podName:8b39dc44-71c0-40cf-8837-dbb9a86be186 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:45.559862807 +0000 UTC m=+104.364182060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/8b39dc44-71c0-40cf-8837-dbb9a86be186-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-swfht" (UID: "8b39dc44-71c0-40cf-8837-dbb9a86be186") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 10 15:43:45.570015 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:45.569971    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8b39dc44-71c0-40cf-8837-dbb9a86be186-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-swfht\" (UID: \"8b39dc44-71c0-40cf-8837-dbb9a86be186\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-swfht"
Sep 10 15:43:45.574011 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:45.573981    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8b39dc44-71c0-40cf-8837-dbb9a86be186-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-swfht\" (UID: \"8b39dc44-71c0-40cf-8837-dbb9a86be186\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-swfht"
Sep 10 15:43:45.652941 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:45.652883    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-swfht"
Sep 10 15:43:46.948695 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:46.948670    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g5l6v"
Sep 10 15:43:47.023824 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:47.023797    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-swfht"]
Sep 10 15:43:47.027416 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:43:47.027387    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b39dc44_71c0_40cf_8837_dbb9a86be186.slice/crio-35ede6c244ab32186064b8316fe9df432bcaf6d574600538878c93f22cf1ba0f WatchSource:0}: Error finding container 35ede6c244ab32186064b8316fe9df432bcaf6d574600538878c93f22cf1ba0f: Status 404 returned error can't find the container with id 35ede6c244ab32186064b8316fe9df432bcaf6d574600538878c93f22cf1ba0f
Sep 10 15:43:47.078582 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:47.078552    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-swfht" event={"ID":"8b39dc44-71c0-40cf-8837-dbb9a86be186","Type":"ContainerStarted","Data":"35ede6c244ab32186064b8316fe9df432bcaf6d574600538878c93f22cf1ba0f"}
Sep 10 15:43:48.085829 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:48.085794    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55549cd8fb-mrpz8" event={"ID":"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899","Type":"ContainerStarted","Data":"885f0b92dc111aeba11bde5d107085370c4b59f6ff32892125d7b49ec44a8b8f"}
Sep 10 15:43:48.105872 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:48.105688    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55549cd8fb-mrpz8" podStartSLOduration=2.628927683 podStartE2EDuration="6.105655957s" totalImagesPullingTime="3.476728274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:48.104638928 +0000 UTC m=+106.908958212" watchObservedRunningTime="2026-09-10 15:43:48.105655957 +0000 UTC m=+106.909975219"
Sep 10 15:43:49.091787 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.091694    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-swfht" event={"ID":"8b39dc44-71c0-40cf-8837-dbb9a86be186","Type":"ContainerStarted","Data":"bb9448db5eef92ffb39a1984976ea6e10403c82ba3bc675879c0d6c4754b2217"}
Sep 10 15:43:49.092267 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.091998    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-swfht"
Sep 10 15:43:49.097231 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.097206    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-swfht"
Sep 10 15:43:49.108343 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.108290    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-swfht" podStartSLOduration=4.375910082 podStartE2EDuration="6.10827273s" totalImagesPullingTime="1.732362648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:49.107420942 +0000 UTC m=+107.911740206" watchObservedRunningTime="2026-09-10 15:43:49.10827273 +0000 UTC m=+107.912591991"
Sep 10 15:43:49.296850 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.296812    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-854d967f57-rmnwh"]
Sep 10 15:43:49.316432 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.316390    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-854d967f57-rmnwh"]
Sep 10 15:43:49.316611 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.316486    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.324511 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.324490    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 15:43:49.407755 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.407671    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-service-ca\") pod \"console-854d967f57-rmnwh\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") " pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.407755 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.407728    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-console-config\") pod \"console-854d967f57-rmnwh\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") " pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.407984 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.407764    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c84f16a2-a8f7-4ca0-8785-6754b9821b17-console-oauth-config\") pod \"console-854d967f57-rmnwh\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") " pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.407984 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.407800    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz75z\" (UniqueName: \"kubernetes.io/projected/c84f16a2-a8f7-4ca0-8785-6754b9821b17-kube-api-access-fz75z\") pod \"console-854d967f57-rmnwh\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") " pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.407984 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.407836    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-oauth-serving-cert\") pod \"console-854d967f57-rmnwh\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") " pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.407984 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.407914    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-trusted-ca-bundle\") pod \"console-854d967f57-rmnwh\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") " pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.407984 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.407941    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c84f16a2-a8f7-4ca0-8785-6754b9821b17-console-serving-cert\") pod \"console-854d967f57-rmnwh\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") " pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.509170 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.509120    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-service-ca\") pod \"console-854d967f57-rmnwh\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") " pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.509363 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.509184    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-console-config\") pod \"console-854d967f57-rmnwh\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") " pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.509363 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.509211    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c84f16a2-a8f7-4ca0-8785-6754b9821b17-console-oauth-config\") pod \"console-854d967f57-rmnwh\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") " pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.509363 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.509237    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz75z\" (UniqueName: \"kubernetes.io/projected/c84f16a2-a8f7-4ca0-8785-6754b9821b17-kube-api-access-fz75z\") pod \"console-854d967f57-rmnwh\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") " pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.509363 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.509261    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-oauth-serving-cert\") pod \"console-854d967f57-rmnwh\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") " pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.509363 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.509305    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-trusted-ca-bundle\") pod \"console-854d967f57-rmnwh\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") " pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.509363 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.509327    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c84f16a2-a8f7-4ca0-8785-6754b9821b17-console-serving-cert\") pod \"console-854d967f57-rmnwh\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") " pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.509954 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.509926    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-service-ca\") pod \"console-854d967f57-rmnwh\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") " pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.510056 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.510008    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-oauth-serving-cert\") pod \"console-854d967f57-rmnwh\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") " pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.510228 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.510209    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-console-config\") pod \"console-854d967f57-rmnwh\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") " pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.510847 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.510805    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-trusted-ca-bundle\") pod \"console-854d967f57-rmnwh\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") " pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.512592 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.512551    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c84f16a2-a8f7-4ca0-8785-6754b9821b17-console-oauth-config\") pod \"console-854d967f57-rmnwh\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") " pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.512773 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.512754    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c84f16a2-a8f7-4ca0-8785-6754b9821b17-console-serving-cert\") pod \"console-854d967f57-rmnwh\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") " pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.518800 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.518775    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz75z\" (UniqueName: \"kubernetes.io/projected/c84f16a2-a8f7-4ca0-8785-6754b9821b17-kube-api-access-fz75z\") pod \"console-854d967f57-rmnwh\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") " pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.627192 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.627136    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:49.944898 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:49.944705    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-854d967f57-rmnwh"]
Sep 10 15:43:49.949834 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:43:49.949778    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84f16a2_a8f7_4ca0_8785_6754b9821b17.slice/crio-5fd02315cece517bc64da27bab029fab107aeba6b489e5afba4f514671b0626d WatchSource:0}: Error finding container 5fd02315cece517bc64da27bab029fab107aeba6b489e5afba4f514671b0626d: Status 404 returned error can't find the container with id 5fd02315cece517bc64da27bab029fab107aeba6b489e5afba4f514671b0626d
Sep 10 15:43:50.099069 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:50.099033    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854d967f57-rmnwh" event={"ID":"c84f16a2-a8f7-4ca0-8785-6754b9821b17","Type":"ContainerStarted","Data":"bcae462cb5dfa254a1491bb5c712d9cf296e3956d04d1331b0e2367e6fa0e643"}
Sep 10 15:43:50.099525 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:50.099089    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854d967f57-rmnwh" event={"ID":"c84f16a2-a8f7-4ca0-8785-6754b9821b17","Type":"ContainerStarted","Data":"5fd02315cece517bc64da27bab029fab107aeba6b489e5afba4f514671b0626d"}
Sep 10 15:43:50.117846 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:50.117753    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-854d967f57-rmnwh" podStartSLOduration=1.117735104 podStartE2EDuration="1.117735104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:50.116661213 +0000 UTC m=+108.920980475" watchObservedRunningTime="2026-09-10 15:43:50.117735104 +0000 UTC m=+108.922054365"
Sep 10 15:43:52.883955 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:52.883917    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:52.884421 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:52.883998    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:43:52.885564 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:52.885531    2564 patch_prober.go:28] interesting pod/console-55549cd8fb-mrpz8 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection refused" start-of-body=
Sep 10 15:43:52.885711 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:52.885589    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-55549cd8fb-mrpz8" podUID="9b1d6ac5-4a56-4c63-aa45-2f65e73bf899" containerName="console" probeResult="failure" output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection refused"
Sep 10 15:43:54.073048 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.072448    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2qrbr"
Sep 10 15:43:54.335568 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.335484    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"]
Sep 10 15:43:54.338370 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.338347    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"
Sep 10 15:43:54.340682 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.340648    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 10 15:43:54.340811 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.340669    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-v8wrm\""
Sep 10 15:43:54.341008 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.340981    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 10 15:43:54.341112 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.341060    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 15:43:54.341112 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.341063    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 10 15:43:54.344930 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.344904    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9l8f4"]
Sep 10 15:43:54.347869 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.347854    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.350224 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.350193    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 15:43:54.350321 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.350307    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 15:43:54.350397 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.350378    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-67ndx\""
Sep 10 15:43:54.350444 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.350390    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 15:43:54.350587 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.350567    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"]
Sep 10 15:43:54.455696 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.455659    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ebaad502-c449-4e86-81b5-3c28d91d99be-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-cvvdl\" (UID: \"ebaad502-c449-4e86-81b5-3c28d91d99be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"
Sep 10 15:43:54.455905 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.455731    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ebaad502-c449-4e86-81b5-3c28d91d99be-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-cvvdl\" (UID: \"ebaad502-c449-4e86-81b5-3c28d91d99be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"
Sep 10 15:43:54.455905 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.455754    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.455905 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.455780    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s8cj\" (UniqueName: \"kubernetes.io/projected/ebaad502-c449-4e86-81b5-3c28d91d99be-kube-api-access-8s8cj\") pod \"kube-state-metrics-74cb57df7-cvvdl\" (UID: \"ebaad502-c449-4e86-81b5-3c28d91d99be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"
Sep 10 15:43:54.455905 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.455806    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebaad502-c449-4e86-81b5-3c28d91d99be-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-cvvdl\" (UID: \"ebaad502-c449-4e86-81b5-3c28d91d99be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"
Sep 10 15:43:54.455905 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.455836    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-node-exporter-accelerators-collector-config\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.455905 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.455859    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-root\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.455905 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.455887    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-sys\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.456294 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.455910    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-node-exporter-wtmp\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.456294 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.455967    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebaad502-c449-4e86-81b5-3c28d91d99be-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-cvvdl\" (UID: \"ebaad502-c449-4e86-81b5-3c28d91d99be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"
Sep 10 15:43:54.456294 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.456032    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-node-exporter-tls\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.456294 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.456058    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqh9j\" (UniqueName: \"kubernetes.io/projected/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-kube-api-access-lqh9j\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.456294 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.456092    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-node-exporter-textfile\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.456294 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.456116    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-metrics-client-ca\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.456294 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.456140    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebaad502-c449-4e86-81b5-3c28d91d99be-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-cvvdl\" (UID: \"ebaad502-c449-4e86-81b5-3c28d91d99be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"
Sep 10 15:43:54.557300 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.556735    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-node-exporter-textfile\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.557300 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.556780    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-metrics-client-ca\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.557300 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.556804    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebaad502-c449-4e86-81b5-3c28d91d99be-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-cvvdl\" (UID: \"ebaad502-c449-4e86-81b5-3c28d91d99be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"
Sep 10 15:43:54.557300 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.556836    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ebaad502-c449-4e86-81b5-3c28d91d99be-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-cvvdl\" (UID: \"ebaad502-c449-4e86-81b5-3c28d91d99be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"
Sep 10 15:43:54.557300 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.556894    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ebaad502-c449-4e86-81b5-3c28d91d99be-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-cvvdl\" (UID: \"ebaad502-c449-4e86-81b5-3c28d91d99be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"
Sep 10 15:43:54.557300 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.556924    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.557300 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:54.556955    2564 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 10 15:43:54.557300 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:43:54.557034    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebaad502-c449-4e86-81b5-3c28d91d99be-kube-state-metrics-tls podName:ebaad502-c449-4e86-81b5-3c28d91d99be nodeName:}" failed. No retries permitted until 2026-09-10 15:43:55.057011357 +0000 UTC m=+113.861330615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ebaad502-c449-4e86-81b5-3c28d91d99be-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-cvvdl" (UID: "ebaad502-c449-4e86-81b5-3c28d91d99be") : secret "kube-state-metrics-tls" not found
Sep 10 15:43:54.557300 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.557089    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-node-exporter-textfile\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.557300 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.556959    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s8cj\" (UniqueName: \"kubernetes.io/projected/ebaad502-c449-4e86-81b5-3c28d91d99be-kube-api-access-8s8cj\") pod \"kube-state-metrics-74cb57df7-cvvdl\" (UID: \"ebaad502-c449-4e86-81b5-3c28d91d99be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"
Sep 10 15:43:54.557300 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.557141    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebaad502-c449-4e86-81b5-3c28d91d99be-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-cvvdl\" (UID: \"ebaad502-c449-4e86-81b5-3c28d91d99be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"
Sep 10 15:43:54.557300 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.557208    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-node-exporter-accelerators-collector-config\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.557300 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.557242    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-root\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.557300 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.557273    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-sys\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.557300 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.557315    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-node-exporter-wtmp\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.558198 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.557350    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebaad502-c449-4e86-81b5-3c28d91d99be-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-cvvdl\" (UID: \"ebaad502-c449-4e86-81b5-3c28d91d99be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"
Sep 10 15:43:54.558198 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.557389    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-node-exporter-tls\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.558198 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.557421    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqh9j\" (UniqueName: \"kubernetes.io/projected/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-kube-api-access-lqh9j\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.558198 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.557429    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-metrics-client-ca\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.558198 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.557509    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-sys\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.558198 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.557621    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-node-exporter-wtmp\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.558198 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.557941    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ebaad502-c449-4e86-81b5-3c28d91d99be-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-cvvdl\" (UID: \"ebaad502-c449-4e86-81b5-3c28d91d99be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"
Sep 10 15:43:54.558198 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.558124    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ebaad502-c449-4e86-81b5-3c28d91d99be-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-cvvdl\" (UID: \"ebaad502-c449-4e86-81b5-3c28d91d99be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"
Sep 10 15:43:54.558587 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.558421    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebaad502-c449-4e86-81b5-3c28d91d99be-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-cvvdl\" (UID: \"ebaad502-c449-4e86-81b5-3c28d91d99be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"
Sep 10 15:43:54.558587 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.558499    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-root\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.558689 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.558627    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-node-exporter-accelerators-collector-config\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.560368 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.560344    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.560866 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.560844    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-node-exporter-tls\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.562566 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.562542    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebaad502-c449-4e86-81b5-3c28d91d99be-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-cvvdl\" (UID: \"ebaad502-c449-4e86-81b5-3c28d91d99be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"
Sep 10 15:43:54.568171 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.568131    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s8cj\" (UniqueName: \"kubernetes.io/projected/ebaad502-c449-4e86-81b5-3c28d91d99be-kube-api-access-8s8cj\") pod \"kube-state-metrics-74cb57df7-cvvdl\" (UID: \"ebaad502-c449-4e86-81b5-3c28d91d99be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"
Sep 10 15:43:54.568592 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.568562    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqh9j\" (UniqueName: \"kubernetes.io/projected/ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb-kube-api-access-lqh9j\") pod \"node-exporter-9l8f4\" (UID: \"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb\") " pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:54.660341 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:54.660257    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9l8f4"
Sep 10 15:43:55.062019 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:55.061976    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebaad502-c449-4e86-81b5-3c28d91d99be-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-cvvdl\" (UID: \"ebaad502-c449-4e86-81b5-3c28d91d99be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"
Sep 10 15:43:55.064897 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:55.064872    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebaad502-c449-4e86-81b5-3c28d91d99be-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-cvvdl\" (UID: \"ebaad502-c449-4e86-81b5-3c28d91d99be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"
Sep 10 15:43:55.251319 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:55.251285    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"
Sep 10 15:43:59.106709 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.106673    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55549cd8fb-mrpz8"]
Sep 10 15:43:59.141808 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.141780    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bcb99f57d-cn2gt"]
Sep 10 15:43:59.145613 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.145590    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.153915 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.153549    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bcb99f57d-cn2gt"]
Sep 10 15:43:59.302173 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.302122    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-service-ca\") pod \"console-7bcb99f57d-cn2gt\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") " pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.302355 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.302204    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq2fk\" (UniqueName: \"kubernetes.io/projected/26757481-2418-47f1-b199-a2e6e4aa6592-kube-api-access-tq2fk\") pod \"console-7bcb99f57d-cn2gt\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") " pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.302355 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.302258    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-oauth-serving-cert\") pod \"console-7bcb99f57d-cn2gt\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") " pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.302355 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.302326    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26757481-2418-47f1-b199-a2e6e4aa6592-console-serving-cert\") pod \"console-7bcb99f57d-cn2gt\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") " pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.302551 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.302393    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-console-config\") pod \"console-7bcb99f57d-cn2gt\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") " pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.302551 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.302419    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-trusted-ca-bundle\") pod \"console-7bcb99f57d-cn2gt\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") " pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.302551 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.302483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26757481-2418-47f1-b199-a2e6e4aa6592-console-oauth-config\") pod \"console-7bcb99f57d-cn2gt\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") " pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.403072 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.402992    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26757481-2418-47f1-b199-a2e6e4aa6592-console-serving-cert\") pod \"console-7bcb99f57d-cn2gt\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") " pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.403280 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.403074    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-console-config\") pod \"console-7bcb99f57d-cn2gt\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") " pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.403280 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.403114    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-trusted-ca-bundle\") pod \"console-7bcb99f57d-cn2gt\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") " pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.403280 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.403194    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26757481-2418-47f1-b199-a2e6e4aa6592-console-oauth-config\") pod \"console-7bcb99f57d-cn2gt\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") " pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.403280 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.403229    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-service-ca\") pod \"console-7bcb99f57d-cn2gt\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") " pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.403280 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.403262    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq2fk\" (UniqueName: \"kubernetes.io/projected/26757481-2418-47f1-b199-a2e6e4aa6592-kube-api-access-tq2fk\") pod \"console-7bcb99f57d-cn2gt\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") " pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.403541 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.403294    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-oauth-serving-cert\") pod \"console-7bcb99f57d-cn2gt\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") " pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.403995 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.403922    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-console-config\") pod \"console-7bcb99f57d-cn2gt\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") " pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.404497 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.404118    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-trusted-ca-bundle\") pod \"console-7bcb99f57d-cn2gt\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") " pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.404497 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.404316    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-oauth-serving-cert\") pod \"console-7bcb99f57d-cn2gt\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") " pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.404657 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.404635    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-service-ca\") pod \"console-7bcb99f57d-cn2gt\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") " pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.405960 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.405939    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26757481-2418-47f1-b199-a2e6e4aa6592-console-oauth-config\") pod \"console-7bcb99f57d-cn2gt\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") " pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.406099 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.406082    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26757481-2418-47f1-b199-a2e6e4aa6592-console-serving-cert\") pod \"console-7bcb99f57d-cn2gt\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") " pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.411746 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.411725    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq2fk\" (UniqueName: \"kubernetes.io/projected/26757481-2418-47f1-b199-a2e6e4aa6592-kube-api-access-tq2fk\") pod \"console-7bcb99f57d-cn2gt\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") " pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.457255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.457209    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:43:59.627846 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.627806    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:59.628029 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.627876    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:43:59.633630 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:43:59.633603    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:44:00.146375 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.146331    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:44:00.424023 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.423994    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bcb99f57d-cn2gt"]
Sep 10 15:44:00.427561 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:44:00.427526    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26757481_2418_47f1_b199_a2e6e4aa6592.slice/crio-90057b04d5845de6c76163008cb33741caf057155c98a20e3cd65e07328dd94d WatchSource:0}: Error finding container 90057b04d5845de6c76163008cb33741caf057155c98a20e3cd65e07328dd94d: Status 404 returned error can't find the container with id 90057b04d5845de6c76163008cb33741caf057155c98a20e3cd65e07328dd94d
Sep 10 15:44:00.447953 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.447915    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl"]
Sep 10 15:44:00.451115 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:44:00.451084    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebaad502_c449_4e86_81b5_3c28d91d99be.slice/crio-1fe40c4605ad1a13212419706f3adcc0d425f7bd76b73e7bb1c2abbf0222e994 WatchSource:0}: Error finding container 1fe40c4605ad1a13212419706f3adcc0d425f7bd76b73e7bb1c2abbf0222e994: Status 404 returned error can't find the container with id 1fe40c4605ad1a13212419706f3adcc0d425f7bd76b73e7bb1c2abbf0222e994
Sep 10 15:44:00.576486 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.575974    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 15:44:00.579744 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.579719    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.583249 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.582901    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sqvfp\""
Sep 10 15:44:00.583249 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.582948    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 15:44:00.583249 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.582968    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 15:44:00.583249 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.583033    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 15:44:00.583249 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.583113    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 15:44:00.583249 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.583190    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 15:44:00.585179 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.584472    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 15:44:00.585179 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.584613    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 15:44:00.585179 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.584983    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 15:44:00.590043 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.585458    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e2aenrpvdd154\""
Sep 10 15:44:00.590043 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.585758    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 15:44:00.590043 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.586405    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 15:44:00.590043 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.586899    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 15:44:00.590043 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.588333    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 15:44:00.590043 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.588682    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 15:44:00.594607 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.594584    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 15:44:00.599529 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.598970    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 15:44:00.601733 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.601606    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 15:44:00.716665 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.716483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-config\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.716665 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.716527    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30f064bd-9318-47a8-bb5b-ad5148b14797-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.716665 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.716555    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30f064bd-9318-47a8-bb5b-ad5148b14797-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.716665 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.716598    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30f064bd-9318-47a8-bb5b-ad5148b14797-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.716665 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.716642    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-web-config\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.717054 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.716715    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.717054 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.716767    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.717054 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.716790    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.717054 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.716814    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.717054 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.716878    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.717054 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.716912    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/30f064bd-9318-47a8-bb5b-ad5148b14797-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.717054 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.716973    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/30f064bd-9318-47a8-bb5b-ad5148b14797-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.717054 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.716997    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/30f064bd-9318-47a8-bb5b-ad5148b14797-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.717054 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.717019    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs9gm\" (UniqueName: \"kubernetes.io/projected/30f064bd-9318-47a8-bb5b-ad5148b14797-kube-api-access-rs9gm\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.717054 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.717047    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.717544 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.717091    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/30f064bd-9318-47a8-bb5b-ad5148b14797-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.717544 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.717119    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30f064bd-9318-47a8-bb5b-ad5148b14797-config-out\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.717544 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.717144    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.717544 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.717200    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30f064bd-9318-47a8-bb5b-ad5148b14797-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.717544 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.717237    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30f064bd-9318-47a8-bb5b-ad5148b14797-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.817945 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.817907    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/30f064bd-9318-47a8-bb5b-ad5148b14797-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.817945 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.817951    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/30f064bd-9318-47a8-bb5b-ad5148b14797-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.818194 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.817976    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs9gm\" (UniqueName: \"kubernetes.io/projected/30f064bd-9318-47a8-bb5b-ad5148b14797-kube-api-access-rs9gm\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.818194 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.818004    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.818194 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.818046    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/30f064bd-9318-47a8-bb5b-ad5148b14797-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.818194 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.818072    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30f064bd-9318-47a8-bb5b-ad5148b14797-config-out\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.818194 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.818095    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.818194 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.818124    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30f064bd-9318-47a8-bb5b-ad5148b14797-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.818487 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.818201    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30f064bd-9318-47a8-bb5b-ad5148b14797-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.818487 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.818252    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-config\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.818487 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.818283    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30f064bd-9318-47a8-bb5b-ad5148b14797-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.818487 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.818306    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30f064bd-9318-47a8-bb5b-ad5148b14797-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.818487 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.818342    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30f064bd-9318-47a8-bb5b-ad5148b14797-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.818487 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.818390    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-web-config\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.818487 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.818425    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.818487 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.818466    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.818805 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.818495    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.818805 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.818519    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.818805 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.818552    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.818805 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.818579    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/30f064bd-9318-47a8-bb5b-ad5148b14797-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.823188 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.821413    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/30f064bd-9318-47a8-bb5b-ad5148b14797-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.823188 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.821426    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30f064bd-9318-47a8-bb5b-ad5148b14797-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.823188 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.821682    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/30f064bd-9318-47a8-bb5b-ad5148b14797-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.823188 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.822008    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30f064bd-9318-47a8-bb5b-ad5148b14797-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.823188 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.822376    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/30f064bd-9318-47a8-bb5b-ad5148b14797-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.823188 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.822904    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/30f064bd-9318-47a8-bb5b-ad5148b14797-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.824425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.823682    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30f064bd-9318-47a8-bb5b-ad5148b14797-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.824425 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.824052    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30f064bd-9318-47a8-bb5b-ad5148b14797-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.825809 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.825783    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-web-config\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.826069 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.826043    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30f064bd-9318-47a8-bb5b-ad5148b14797-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.828420 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.827930    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30f064bd-9318-47a8-bb5b-ad5148b14797-config-out\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.828420 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.827980    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.828420 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.828143    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-config\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.828744 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.828721    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.829471 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.829432    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.830301 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.830276    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.830518 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.830461    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.830931 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.830906    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.831009 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.830933    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30f064bd-9318-47a8-bb5b-ad5148b14797-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.832275 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.832241    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs9gm\" (UniqueName: \"kubernetes.io/projected/30f064bd-9318-47a8-bb5b-ad5148b14797-kube-api-access-rs9gm\") pod \"prometheus-k8s-0\" (UID: \"30f064bd-9318-47a8-bb5b-ad5148b14797\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:00.899652 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:00.899339    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:01.149743 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:01.149698    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9l8f4" event={"ID":"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb","Type":"ContainerStarted","Data":"096e1ebee9c09acdba5b245e10e26d1ff9d55c20b62eeda31511154a00e6ea77"}
Sep 10 15:44:01.151338 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:01.151305    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl" event={"ID":"ebaad502-c449-4e86-81b5-3c28d91d99be","Type":"ContainerStarted","Data":"1fe40c4605ad1a13212419706f3adcc0d425f7bd76b73e7bb1c2abbf0222e994"}
Sep 10 15:44:01.156775 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:01.156739    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcb99f57d-cn2gt" event={"ID":"26757481-2418-47f1-b199-a2e6e4aa6592","Type":"ContainerStarted","Data":"132331f42d6d958b1fe83493ec6c8b34ae594251dfee038264bd520443c8c55b"}
Sep 10 15:44:01.156940 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:01.156797    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcb99f57d-cn2gt" event={"ID":"26757481-2418-47f1-b199-a2e6e4aa6592","Type":"ContainerStarted","Data":"90057b04d5845de6c76163008cb33741caf057155c98a20e3cd65e07328dd94d"}
Sep 10 15:44:01.162215 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:01.162184    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-qlzmm" event={"ID":"b8a041bb-e51d-4b49-92c8-e25b58a05f3a","Type":"ContainerStarted","Data":"06cf65268c9cb475e756020a627c6a9d1f91f9bf4d0349d47d433e96c9710f63"}
Sep 10 15:44:01.162347 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:01.162239    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-qlzmm"
Sep 10 15:44:01.180465 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:01.179765    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bcb99f57d-cn2gt" podStartSLOduration=2.179742825 podStartE2EDuration="2.179742825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:44:01.177114892 +0000 UTC m=+119.981434150" watchObservedRunningTime="2026-09-10 15:44:01.179742825 +0000 UTC m=+119.984062088"
Sep 10 15:44:01.182217 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:01.181741    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-qlzmm"
Sep 10 15:44:01.213730 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:01.213197    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-qlzmm" podStartSLOduration=2.501320781 podStartE2EDuration="19.21315206s" totalImagesPullingTime="16.711831279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:44:01.19829842 +0000 UTC m=+120.002617683" watchObservedRunningTime="2026-09-10 15:44:01.21315206 +0000 UTC m=+120.017471322"
Sep 10 15:44:01.213730 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:01.213642    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 15:44:02.166550 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:02.166507    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30f064bd-9318-47a8-bb5b-ad5148b14797","Type":"ContainerStarted","Data":"d78cf50929d763a8174f06b9d8042b993d55e617d43e1a3897b839b0a53ffada"}
Sep 10 15:44:03.172898 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:03.172856    2564 generic.go:356] "Generic (PLEG): container finished" podID="ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb" containerID="9293159451c2dd466fe2baace128382740017cb620c317516e53161b73fc211f" exitCode=0
Sep 10 15:44:03.173400 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:03.172946    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9l8f4" event={"ID":"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb","Type":"ContainerDied","Data":"9293159451c2dd466fe2baace128382740017cb620c317516e53161b73fc211f"}
Sep 10 15:44:03.175685 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:03.175658    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl" event={"ID":"ebaad502-c449-4e86-81b5-3c28d91d99be","Type":"ContainerStarted","Data":"dd565b4e2306fafe11232294c95936cc44f2f9bc509c0d4e244f87b76ca9aa2a"}
Sep 10 15:44:03.175813 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:03.175720    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl" event={"ID":"ebaad502-c449-4e86-81b5-3c28d91d99be","Type":"ContainerStarted","Data":"6bf4b03883d97cc5b7a5418d8ae64c6a010fa89e69477c7f2e6070ca6a4c24c7"}
Sep 10 15:44:03.175813 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:03.175750    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl" event={"ID":"ebaad502-c449-4e86-81b5-3c28d91d99be","Type":"ContainerStarted","Data":"f5277e9e52c78ada6a6524aabdd4ab6ccfcf6891f77006fde9b09888a0ca9972"}
Sep 10 15:44:03.212629 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:03.212574    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-cvvdl" podStartSLOduration=7.26719835 podStartE2EDuration="9.212558844s" totalImagesPullingTime="1.945360494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:44:03.211480073 +0000 UTC m=+122.015799335" watchObservedRunningTime="2026-09-10 15:44:03.212558844 +0000 UTC m=+122.016878103"
Sep 10 15:44:04.183100 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:04.183055    2564 generic.go:356] "Generic (PLEG): container finished" podID="30f064bd-9318-47a8-bb5b-ad5148b14797" containerID="c356178c1375622373e5165ea65c6384775f41e76f84d0177935d5395f4eb868" exitCode=0
Sep 10 15:44:04.183534 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:04.183148    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30f064bd-9318-47a8-bb5b-ad5148b14797","Type":"ContainerDied","Data":"c356178c1375622373e5165ea65c6384775f41e76f84d0177935d5395f4eb868"}
Sep 10 15:44:04.186554 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:04.186523    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9l8f4" event={"ID":"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb","Type":"ContainerStarted","Data":"c935577f6b09e545aa53a3b6f89db793d4a10b194b077f4a5217ac6e1c01a976"}
Sep 10 15:44:04.186663 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:04.186584    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9l8f4" event={"ID":"ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb","Type":"ContainerStarted","Data":"a120157a7b25b3db29c3d7931d7a0f7270043123223257ce672ffaf87ecbe9ac"}
Sep 10 15:44:04.230378 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:04.230317    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9l8f4" podStartSLOduration=8.232395394 podStartE2EDuration="10.230295289s" totalImagesPullingTime="1.899773685s" totalInitContainerRuntime="98.12621ms" isStatefulPod=true podCreationTimestamp="2026-09-10 15:43:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:44:04.229538057 +0000 UTC m=+123.033857332" watchObservedRunningTime="2026-09-10 15:44:04.230295289 +0000 UTC m=+123.034614551"
Sep 10 15:44:05.074073 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:05.074041    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cf899df5c-trxv8"
Sep 10 15:44:09.217664 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:09.217628    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30f064bd-9318-47a8-bb5b-ad5148b14797","Type":"ContainerStarted","Data":"284071601aaa203c59d8a55dcdb7512c7695936d0c3072c8fd08d16bbe733faa"}
Sep 10 15:44:09.218178 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:09.217711    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30f064bd-9318-47a8-bb5b-ad5148b14797","Type":"ContainerStarted","Data":"9b9768557d6d7136ab803e9ebc9510dcfde8a026a396898ae5501326ca45f0ad"}
Sep 10 15:44:09.458087 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:09.458044    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:44:09.458300 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:09.458118    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:44:09.464709 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:09.464681    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:44:10.226240 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:10.226208    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:44:10.274247 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:10.274150    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-854d967f57-rmnwh"]
Sep 10 15:44:11.228530 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:11.228503    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30f064bd-9318-47a8-bb5b-ad5148b14797","Type":"ContainerStarted","Data":"e633478a4583d4fb6339e8f4a8f020ae195fc7fba28bc16a1ad20c90b18b5ac3"}
Sep 10 15:44:11.228864 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:11.228577    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30f064bd-9318-47a8-bb5b-ad5148b14797","Type":"ContainerStarted","Data":"b273e265e9f1df3feba13dba1d92ad332b440a04b9e26866f09e3bb5bc250964"}
Sep 10 15:44:11.228864 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:11.228618    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30f064bd-9318-47a8-bb5b-ad5148b14797","Type":"ContainerStarted","Data":"e592425f849149c367fbb142a43d289a86705a7157029d1da94bede199030ad4"}
Sep 10 15:44:11.436058 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:11.436027    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs\") pod \"network-metrics-daemon-m8bvz\" (UID: \"a140ffcb-b59c-4e55-95e8-e9ed589396b1\") " pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:44:11.438569 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:11.438547    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a140ffcb-b59c-4e55-95e8-e9ed589396b1-metrics-certs\") pod \"network-metrics-daemon-m8bvz\" (UID: \"a140ffcb-b59c-4e55-95e8-e9ed589396b1\") " pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:44:11.487040 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:11.487012    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b79h8\""
Sep 10 15:44:11.495244 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:11.495212    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m8bvz"
Sep 10 15:44:11.778818 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:11.778783    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m8bvz"]
Sep 10 15:44:11.780909 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:44:11.780878    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda140ffcb_b59c_4e55_95e8_e9ed589396b1.slice/crio-7534f74b1107780ea8835d970a096aecbc560a5fab32f195097727101f752546 WatchSource:0}: Error finding container 7534f74b1107780ea8835d970a096aecbc560a5fab32f195097727101f752546: Status 404 returned error can't find the container with id 7534f74b1107780ea8835d970a096aecbc560a5fab32f195097727101f752546
Sep 10 15:44:12.235580 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:12.235545    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30f064bd-9318-47a8-bb5b-ad5148b14797","Type":"ContainerStarted","Data":"dc838b6f0e9e36f088175cdcaf10c8f4a2a75e23062ee7489a1c9c5ca2ec951c"}
Sep 10 15:44:12.236772 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:12.236746    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m8bvz" event={"ID":"a140ffcb-b59c-4e55-95e8-e9ed589396b1","Type":"ContainerStarted","Data":"7534f74b1107780ea8835d970a096aecbc560a5fab32f195097727101f752546"}
Sep 10 15:44:12.266843 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:12.266787    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.745957906 podStartE2EDuration="12.266769767s" totalImagesPullingTime="8.496661989s" totalInitContainerRuntime="24.149872ms" isStatefulPod=false podCreationTimestamp="2026-09-10 15:44:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:44:12.262178674 +0000 UTC m=+131.066497926" watchObservedRunningTime="2026-09-10 15:44:12.266769767 +0000 UTC m=+131.071089060"
Sep 10 15:44:14.247711 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:14.247676    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m8bvz" event={"ID":"a140ffcb-b59c-4e55-95e8-e9ed589396b1","Type":"ContainerStarted","Data":"9c969c7e3721d601b32cd11d5ab1d88f9c22a88c800e24374a6f22b816f1f370"}
Sep 10 15:44:14.248222 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:14.247731    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m8bvz" event={"ID":"a140ffcb-b59c-4e55-95e8-e9ed589396b1","Type":"ContainerStarted","Data":"e2ac7345d3f3e6f9b7df0eaf8f403b6ad83dafdf4bca12c44fa0707fb68846ca"}
Sep 10 15:44:14.266006 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:14.265938    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-m8bvz" podStartSLOduration=130.505521431 podStartE2EDuration="2m12.265917932s" totalImagesPullingTime="1.760396501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:44:14.265716892 +0000 UTC m=+133.070036137" watchObservedRunningTime="2026-09-10 15:44:14.265917932 +0000 UTC m=+133.070237193"
Sep 10 15:44:15.900548 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:15.900510    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:44:21.276780 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:21.276742    2564 generic.go:356] "Generic (PLEG): container finished" podID="21f9ed82-8d7f-48dc-af6f-a22d73178304" containerID="42fd9dc993506e93d41e9058c2c5b6a9804bd5c7b5e2771750ba07faf85c88f6" exitCode=0
Sep 10 15:44:21.277191 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:21.276816    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn" event={"ID":"21f9ed82-8d7f-48dc-af6f-a22d73178304","Type":"ContainerDied","Data":"42fd9dc993506e93d41e9058c2c5b6a9804bd5c7b5e2771750ba07faf85c88f6"}
Sep 10 15:44:21.277238 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:21.277218    2564 scope.go:117] "RemoveContainer" containerID="42fd9dc993506e93d41e9058c2c5b6a9804bd5c7b5e2771750ba07faf85c88f6"
Sep 10 15:44:22.283380 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:22.283344    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-kk7mn" event={"ID":"21f9ed82-8d7f-48dc-af6f-a22d73178304","Type":"ContainerStarted","Data":"5259342e6d28dda63e1c4d11958e2a113ddcec799a854eee8eafe55525d637a0"}
Sep 10 15:44:24.131191 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.131115    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55549cd8fb-mrpz8" podUID="9b1d6ac5-4a56-4c63-aa45-2f65e73bf899" containerName="console" containerID="cri-o://885f0b92dc111aeba11bde5d107085370c4b59f6ff32892125d7b49ec44a8b8f" gracePeriod=15
Sep 10 15:44:24.294597 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.294564    2564 generic.go:356] "Generic (PLEG): container finished" podID="9b1d6ac5-4a56-4c63-aa45-2f65e73bf899" containerID="885f0b92dc111aeba11bde5d107085370c4b59f6ff32892125d7b49ec44a8b8f" exitCode=0
Sep 10 15:44:24.294597 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.294589    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55549cd8fb-mrpz8" event={"ID":"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899","Type":"ContainerDied","Data":"885f0b92dc111aeba11bde5d107085370c4b59f6ff32892125d7b49ec44a8b8f"}
Sep 10 15:44:24.412953 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.412874    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:44:24.556395 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.556359    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-service-ca\") pod \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") "
Sep 10 15:44:24.556587 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.556404    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-console-serving-cert\") pod \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") "
Sep 10 15:44:24.556587 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.556479    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkx7t\" (UniqueName: \"kubernetes.io/projected/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-kube-api-access-bkx7t\") pod \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") "
Sep 10 15:44:24.556587 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.556547    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-console-oauth-config\") pod \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") "
Sep 10 15:44:24.556746 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.556589    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-oauth-serving-cert\") pod \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") "
Sep 10 15:44:24.556746 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.556644    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-console-config\") pod \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\" (UID: \"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899\") "
Sep 10 15:44:24.557024 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.556903    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-service-ca" (OuterVolumeSpecName: "service-ca") pod "9b1d6ac5-4a56-4c63-aa45-2f65e73bf899" (UID: "9b1d6ac5-4a56-4c63-aa45-2f65e73bf899"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:44:24.557024 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.556990    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9b1d6ac5-4a56-4c63-aa45-2f65e73bf899" (UID: "9b1d6ac5-4a56-4c63-aa45-2f65e73bf899"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:44:24.557193 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.557139    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-oauth-serving-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:44:24.557193 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.557152    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:44:24.557516 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.557438    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-console-config" (OuterVolumeSpecName: "console-config") pod "9b1d6ac5-4a56-4c63-aa45-2f65e73bf899" (UID: "9b1d6ac5-4a56-4c63-aa45-2f65e73bf899"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:44:24.559031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.559001    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-kube-api-access-bkx7t" (OuterVolumeSpecName: "kube-api-access-bkx7t") pod "9b1d6ac5-4a56-4c63-aa45-2f65e73bf899" (UID: "9b1d6ac5-4a56-4c63-aa45-2f65e73bf899"). InnerVolumeSpecName "kube-api-access-bkx7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:44:24.559031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.559019    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9b1d6ac5-4a56-4c63-aa45-2f65e73bf899" (UID: "9b1d6ac5-4a56-4c63-aa45-2f65e73bf899"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:44:24.559272 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.559251    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9b1d6ac5-4a56-4c63-aa45-2f65e73bf899" (UID: "9b1d6ac5-4a56-4c63-aa45-2f65e73bf899"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:44:24.658247 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.658212    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-console-serving-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:44:24.658247 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.658241    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkx7t\" (UniqueName: \"kubernetes.io/projected/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-kube-api-access-bkx7t\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:44:24.658247 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.658252    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-console-oauth-config\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:44:24.658471 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:24.658260    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899-console-config\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:44:25.300679 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:25.300644    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55549cd8fb-mrpz8" event={"ID":"9b1d6ac5-4a56-4c63-aa45-2f65e73bf899","Type":"ContainerDied","Data":"003c9a2ebcb47767fa023e97fc5dce6783b1957ab09ecf49f1ffc40e7696d32d"}
Sep 10 15:44:25.301094 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:25.300658    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55549cd8fb-mrpz8"
Sep 10 15:44:25.301094 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:25.300729    2564 scope.go:117] "RemoveContainer" containerID="885f0b92dc111aeba11bde5d107085370c4b59f6ff32892125d7b49ec44a8b8f"
Sep 10 15:44:25.322463 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:25.322438    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55549cd8fb-mrpz8"]
Sep 10 15:44:25.327562 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:25.327527    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55549cd8fb-mrpz8"]
Sep 10 15:44:25.671743 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:25.671714    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b1d6ac5-4a56-4c63-aa45-2f65e73bf899" path="/var/lib/kubelet/pods/9b1d6ac5-4a56-4c63-aa45-2f65e73bf899/volumes"
Sep 10 15:44:35.302596 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.302542    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-854d967f57-rmnwh" podUID="c84f16a2-a8f7-4ca0-8785-6754b9821b17" containerName="console" containerID="cri-o://bcae462cb5dfa254a1491bb5c712d9cf296e3956d04d1331b0e2367e6fa0e643" gracePeriod=15
Sep 10 15:44:35.593132 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.593101    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:44:35.756826 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.756788    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c84f16a2-a8f7-4ca0-8785-6754b9821b17-console-serving-cert\") pod \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") "
Sep 10 15:44:35.757022 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.756835    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-service-ca\") pod \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") "
Sep 10 15:44:35.757022 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.756874    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-oauth-serving-cert\") pod \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") "
Sep 10 15:44:35.757022 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.756909    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c84f16a2-a8f7-4ca0-8785-6754b9821b17-console-oauth-config\") pod \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") "
Sep 10 15:44:35.757022 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.756983    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-trusted-ca-bundle\") pod \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") "
Sep 10 15:44:35.757022 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.757018    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz75z\" (UniqueName: \"kubernetes.io/projected/c84f16a2-a8f7-4ca0-8785-6754b9821b17-kube-api-access-fz75z\") pod \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") "
Sep 10 15:44:35.757303 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.757051    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-console-config\") pod \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\" (UID: \"c84f16a2-a8f7-4ca0-8785-6754b9821b17\") "
Sep 10 15:44:35.757370 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.757347    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c84f16a2-a8f7-4ca0-8785-6754b9821b17" (UID: "c84f16a2-a8f7-4ca0-8785-6754b9821b17"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:44:35.757422 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.757386    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-service-ca" (OuterVolumeSpecName: "service-ca") pod "c84f16a2-a8f7-4ca0-8785-6754b9821b17" (UID: "c84f16a2-a8f7-4ca0-8785-6754b9821b17"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:44:35.757485 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.757450    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c84f16a2-a8f7-4ca0-8785-6754b9821b17" (UID: "c84f16a2-a8f7-4ca0-8785-6754b9821b17"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:44:35.757781 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.757755    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-console-config" (OuterVolumeSpecName: "console-config") pod "c84f16a2-a8f7-4ca0-8785-6754b9821b17" (UID: "c84f16a2-a8f7-4ca0-8785-6754b9821b17"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:44:35.759202 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.759177    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c84f16a2-a8f7-4ca0-8785-6754b9821b17-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c84f16a2-a8f7-4ca0-8785-6754b9821b17" (UID: "c84f16a2-a8f7-4ca0-8785-6754b9821b17"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:44:35.759323 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.759297    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84f16a2-a8f7-4ca0-8785-6754b9821b17-kube-api-access-fz75z" (OuterVolumeSpecName: "kube-api-access-fz75z") pod "c84f16a2-a8f7-4ca0-8785-6754b9821b17" (UID: "c84f16a2-a8f7-4ca0-8785-6754b9821b17"). InnerVolumeSpecName "kube-api-access-fz75z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:44:35.759368 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.759324    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c84f16a2-a8f7-4ca0-8785-6754b9821b17-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c84f16a2-a8f7-4ca0-8785-6754b9821b17" (UID: "c84f16a2-a8f7-4ca0-8785-6754b9821b17"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:44:35.857883 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.857802    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-trusted-ca-bundle\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:44:35.857883 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.857847    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz75z\" (UniqueName: \"kubernetes.io/projected/c84f16a2-a8f7-4ca0-8785-6754b9821b17-kube-api-access-fz75z\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:44:35.857883 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.857860    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-console-config\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:44:35.857883 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.857870    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c84f16a2-a8f7-4ca0-8785-6754b9821b17-console-serving-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:44:35.857883 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.857878    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:44:35.857883 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.857885    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c84f16a2-a8f7-4ca0-8785-6754b9821b17-oauth-serving-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:44:35.858186 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:35.857893    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c84f16a2-a8f7-4ca0-8785-6754b9821b17-console-oauth-config\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:44:36.347264 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:36.347227    2564 generic.go:356] "Generic (PLEG): container finished" podID="c84f16a2-a8f7-4ca0-8785-6754b9821b17" containerID="bcae462cb5dfa254a1491bb5c712d9cf296e3956d04d1331b0e2367e6fa0e643" exitCode=0
Sep 10 15:44:36.347764 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:36.347295    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854d967f57-rmnwh"
Sep 10 15:44:36.347764 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:36.347313    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854d967f57-rmnwh" event={"ID":"c84f16a2-a8f7-4ca0-8785-6754b9821b17","Type":"ContainerDied","Data":"bcae462cb5dfa254a1491bb5c712d9cf296e3956d04d1331b0e2367e6fa0e643"}
Sep 10 15:44:36.347764 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:36.347371    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854d967f57-rmnwh" event={"ID":"c84f16a2-a8f7-4ca0-8785-6754b9821b17","Type":"ContainerDied","Data":"5fd02315cece517bc64da27bab029fab107aeba6b489e5afba4f514671b0626d"}
Sep 10 15:44:36.347764 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:36.347401    2564 scope.go:117] "RemoveContainer" containerID="bcae462cb5dfa254a1491bb5c712d9cf296e3956d04d1331b0e2367e6fa0e643"
Sep 10 15:44:36.356763 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:36.356747    2564 scope.go:117] "RemoveContainer" containerID="bcae462cb5dfa254a1491bb5c712d9cf296e3956d04d1331b0e2367e6fa0e643"
Sep 10 15:44:36.357045 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:44:36.357025    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcae462cb5dfa254a1491bb5c712d9cf296e3956d04d1331b0e2367e6fa0e643\": container with ID starting with bcae462cb5dfa254a1491bb5c712d9cf296e3956d04d1331b0e2367e6fa0e643 not found: ID does not exist" containerID="bcae462cb5dfa254a1491bb5c712d9cf296e3956d04d1331b0e2367e6fa0e643"
Sep 10 15:44:36.357091 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:36.357053    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcae462cb5dfa254a1491bb5c712d9cf296e3956d04d1331b0e2367e6fa0e643"} err="failed to get container status \"bcae462cb5dfa254a1491bb5c712d9cf296e3956d04d1331b0e2367e6fa0e643\": rpc error: code = NotFound desc = could not find container \"bcae462cb5dfa254a1491bb5c712d9cf296e3956d04d1331b0e2367e6fa0e643\": container with ID starting with bcae462cb5dfa254a1491bb5c712d9cf296e3956d04d1331b0e2367e6fa0e643 not found: ID does not exist"
Sep 10 15:44:36.372525 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:36.372505    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-854d967f57-rmnwh"]
Sep 10 15:44:36.377630 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:36.377612    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-854d967f57-rmnwh"]
Sep 10 15:44:37.670813 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:37.670778    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c84f16a2-a8f7-4ca0-8785-6754b9821b17" path="/var/lib/kubelet/pods/c84f16a2-a8f7-4ca0-8785-6754b9821b17/volumes"
Sep 10 15:44:51.409768 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:51.409731    2564 generic.go:356] "Generic (PLEG): container finished" podID="811b2e5a-e4b6-45d9-99d4-4669a476afbc" containerID="3f007b1cd3b21f5a16356b7b7cbdd4fd44fec2218b5391289bb9bc011bb28155" exitCode=0
Sep 10 15:44:51.410261 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:51.409795    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-nsnkp" event={"ID":"811b2e5a-e4b6-45d9-99d4-4669a476afbc","Type":"ContainerDied","Data":"3f007b1cd3b21f5a16356b7b7cbdd4fd44fec2218b5391289bb9bc011bb28155"}
Sep 10 15:44:51.410261 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:51.410120    2564 scope.go:117] "RemoveContainer" containerID="3f007b1cd3b21f5a16356b7b7cbdd4fd44fec2218b5391289bb9bc011bb28155"
Sep 10 15:44:52.415887 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:44:52.415849    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-nsnkp" event={"ID":"811b2e5a-e4b6-45d9-99d4-4669a476afbc","Type":"ContainerStarted","Data":"2fccd773497959b5a0c8ef9290ab1c4a65a116a672b03fd883bc46b0d2bc3f8d"}
Sep 10 15:45:00.900133 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:00.900077    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:45:00.919880 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:00.919857    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:45:01.466376 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:01.466348    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 15:45:22.429484 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.429445    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66bc9588f5-k64z2"]
Sep 10 15:45:22.429959 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.429882    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b1d6ac5-4a56-4c63-aa45-2f65e73bf899" containerName="console"
Sep 10 15:45:22.429959 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.429901    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b1d6ac5-4a56-4c63-aa45-2f65e73bf899" containerName="console"
Sep 10 15:45:22.429959 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.429927    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c84f16a2-a8f7-4ca0-8785-6754b9821b17" containerName="console"
Sep 10 15:45:22.429959 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.429936    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c84f16a2-a8f7-4ca0-8785-6754b9821b17" containerName="console"
Sep 10 15:45:22.430178 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.430023    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c84f16a2-a8f7-4ca0-8785-6754b9821b17" containerName="console"
Sep 10 15:45:22.430178 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.430038    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b1d6ac5-4a56-4c63-aa45-2f65e73bf899" containerName="console"
Sep 10 15:45:22.431966 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.431947    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.443790 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.443764    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66bc9588f5-k64z2"]
Sep 10 15:45:22.566470 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.566427    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-console-config\") pod \"console-66bc9588f5-k64z2\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") " pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.566470 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.566473    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e280aab7-7513-492f-96c3-7361f6058a99-console-oauth-config\") pod \"console-66bc9588f5-k64z2\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") " pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.566676 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.566521    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-trusted-ca-bundle\") pod \"console-66bc9588f5-k64z2\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") " pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.566676 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.566564    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e280aab7-7513-492f-96c3-7361f6058a99-console-serving-cert\") pod \"console-66bc9588f5-k64z2\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") " pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.566676 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.566640    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-service-ca\") pod \"console-66bc9588f5-k64z2\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") " pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.566676 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.566670    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-oauth-serving-cert\") pod \"console-66bc9588f5-k64z2\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") " pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.566805 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.566689    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4psb\" (UniqueName: \"kubernetes.io/projected/e280aab7-7513-492f-96c3-7361f6058a99-kube-api-access-x4psb\") pod \"console-66bc9588f5-k64z2\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") " pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.667806 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.667769    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-service-ca\") pod \"console-66bc9588f5-k64z2\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") " pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.667806 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.667803    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-oauth-serving-cert\") pod \"console-66bc9588f5-k64z2\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") " pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.668043 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.667820    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4psb\" (UniqueName: \"kubernetes.io/projected/e280aab7-7513-492f-96c3-7361f6058a99-kube-api-access-x4psb\") pod \"console-66bc9588f5-k64z2\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") " pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.668043 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.667849    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-console-config\") pod \"console-66bc9588f5-k64z2\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") " pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.668043 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.667866    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e280aab7-7513-492f-96c3-7361f6058a99-console-oauth-config\") pod \"console-66bc9588f5-k64z2\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") " pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.668043 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.667895    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-trusted-ca-bundle\") pod \"console-66bc9588f5-k64z2\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") " pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.668043 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.667946    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e280aab7-7513-492f-96c3-7361f6058a99-console-serving-cert\") pod \"console-66bc9588f5-k64z2\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") " pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.668614 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.668588    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-oauth-serving-cert\") pod \"console-66bc9588f5-k64z2\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") " pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.668715 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.668588    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-service-ca\") pod \"console-66bc9588f5-k64z2\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") " pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.668715 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.668631    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-console-config\") pod \"console-66bc9588f5-k64z2\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") " pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.668878 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.668849    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-trusted-ca-bundle\") pod \"console-66bc9588f5-k64z2\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") " pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.670546 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.670526    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e280aab7-7513-492f-96c3-7361f6058a99-console-oauth-config\") pod \"console-66bc9588f5-k64z2\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") " pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.670625 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.670604    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e280aab7-7513-492f-96c3-7361f6058a99-console-serving-cert\") pod \"console-66bc9588f5-k64z2\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") " pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.676082 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.676063    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4psb\" (UniqueName: \"kubernetes.io/projected/e280aab7-7513-492f-96c3-7361f6058a99-kube-api-access-x4psb\") pod \"console-66bc9588f5-k64z2\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") " pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:22.742877 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:22.742782    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:23.014595 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:23.014506    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66bc9588f5-k64z2"]
Sep 10 15:45:23.018620 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:45:23.018593    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode280aab7_7513_492f_96c3_7361f6058a99.slice/crio-4377f99fb31645863152929d141a350ac7a8f3438630d2efa80b93d8b7520435 WatchSource:0}: Error finding container 4377f99fb31645863152929d141a350ac7a8f3438630d2efa80b93d8b7520435: Status 404 returned error can't find the container with id 4377f99fb31645863152929d141a350ac7a8f3438630d2efa80b93d8b7520435
Sep 10 15:45:23.542039 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:23.541998    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66bc9588f5-k64z2" event={"ID":"e280aab7-7513-492f-96c3-7361f6058a99","Type":"ContainerStarted","Data":"9a2cd9f3c7ef6434c030dbc566482b92f2fa2110132d27a63c88e8aa995f0209"}
Sep 10 15:45:23.542447 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:23.542082    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66bc9588f5-k64z2" event={"ID":"e280aab7-7513-492f-96c3-7361f6058a99","Type":"ContainerStarted","Data":"4377f99fb31645863152929d141a350ac7a8f3438630d2efa80b93d8b7520435"}
Sep 10 15:45:23.561096 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:23.561044    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66bc9588f5-k64z2" podStartSLOduration=1.5610287139999999 podStartE2EDuration="1.561028714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:23.55908413 +0000 UTC m=+202.363403390" watchObservedRunningTime="2026-09-10 15:45:23.561028714 +0000 UTC m=+202.365347974"
Sep 10 15:45:30.411090 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.411041    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66bc9588f5-k64z2"]
Sep 10 15:45:30.435754 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.435719    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-678d855bc7-cgqgg"]
Sep 10 15:45:30.438395 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.438376    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.451537 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.451513    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-678d855bc7-cgqgg"]
Sep 10 15:45:30.541579 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.541541    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-service-ca\") pod \"console-678d855bc7-cgqgg\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") " pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.541761 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.541599    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-oauth-serving-cert\") pod \"console-678d855bc7-cgqgg\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") " pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.541761 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.541636    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-console-oauth-config\") pod \"console-678d855bc7-cgqgg\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") " pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.541761 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.541662    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-console-config\") pod \"console-678d855bc7-cgqgg\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") " pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.541761 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.541681    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sskfw\" (UniqueName: \"kubernetes.io/projected/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-kube-api-access-sskfw\") pod \"console-678d855bc7-cgqgg\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") " pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.541761 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.541715    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-trusted-ca-bundle\") pod \"console-678d855bc7-cgqgg\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") " pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.541761 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.541739    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-console-serving-cert\") pod \"console-678d855bc7-cgqgg\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") " pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.642944 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.642909    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sskfw\" (UniqueName: \"kubernetes.io/projected/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-kube-api-access-sskfw\") pod \"console-678d855bc7-cgqgg\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") " pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.643123 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.642953    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-trusted-ca-bundle\") pod \"console-678d855bc7-cgqgg\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") " pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.643123 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.642976    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-console-serving-cert\") pod \"console-678d855bc7-cgqgg\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") " pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.643123 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.643058    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-service-ca\") pod \"console-678d855bc7-cgqgg\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") " pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.643123 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.643110    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-oauth-serving-cert\") pod \"console-678d855bc7-cgqgg\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") " pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.643371 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.643134    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-console-oauth-config\") pod \"console-678d855bc7-cgqgg\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") " pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.643371 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.643192    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-console-config\") pod \"console-678d855bc7-cgqgg\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") " pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.643894 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.643865    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-service-ca\") pod \"console-678d855bc7-cgqgg\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") " pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.644010 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.643864    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-oauth-serving-cert\") pod \"console-678d855bc7-cgqgg\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") " pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.644010 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.643928    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-console-config\") pod \"console-678d855bc7-cgqgg\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") " pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.644362 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.644337    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-trusted-ca-bundle\") pod \"console-678d855bc7-cgqgg\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") " pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.645674 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.645644    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-console-serving-cert\") pod \"console-678d855bc7-cgqgg\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") " pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.645754 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.645728    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-console-oauth-config\") pod \"console-678d855bc7-cgqgg\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") " pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.653091 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.653072    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sskfw\" (UniqueName: \"kubernetes.io/projected/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-kube-api-access-sskfw\") pod \"console-678d855bc7-cgqgg\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") " pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:30.748518 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:30.748476    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:31.021239 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:31.021216    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-678d855bc7-cgqgg"]
Sep 10 15:45:31.023125 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:45:31.023096    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7311dae7_6fd9_4e55_91b9_0110c33ff8a8.slice/crio-98234071a5711512b6c6d05f97b33f3eebf013b1ed467eb7c75ad59f3018a307 WatchSource:0}: Error finding container 98234071a5711512b6c6d05f97b33f3eebf013b1ed467eb7c75ad59f3018a307: Status 404 returned error can't find the container with id 98234071a5711512b6c6d05f97b33f3eebf013b1ed467eb7c75ad59f3018a307
Sep 10 15:45:31.576244 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:31.576194    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678d855bc7-cgqgg" event={"ID":"7311dae7-6fd9-4e55-91b9-0110c33ff8a8","Type":"ContainerStarted","Data":"515ff79d2ec53d4d45e9979fff5fa4451f279f5282bbd5afe0cab2498d8d1b02"}
Sep 10 15:45:31.576640 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:31.576264    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678d855bc7-cgqgg" event={"ID":"7311dae7-6fd9-4e55-91b9-0110c33ff8a8","Type":"ContainerStarted","Data":"98234071a5711512b6c6d05f97b33f3eebf013b1ed467eb7c75ad59f3018a307"}
Sep 10 15:45:31.596494 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:31.596449    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-678d855bc7-cgqgg" podStartSLOduration=1.596432901 podStartE2EDuration="1.596432901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:31.595108008 +0000 UTC m=+210.399427268" watchObservedRunningTime="2026-09-10 15:45:31.596432901 +0000 UTC m=+210.400752162"
Sep 10 15:45:32.743597 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:32.743548    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:40.749376 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:40.749325    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:40.749376 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:40.749393    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:40.759256 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:40.758534    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:41.619570 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:41.619538    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 15:45:41.665642 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:41.665601    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bcb99f57d-cn2gt"]
Sep 10 15:45:55.434110 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.434048    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66bc9588f5-k64z2" podUID="e280aab7-7513-492f-96c3-7361f6058a99" containerName="console" containerID="cri-o://9a2cd9f3c7ef6434c030dbc566482b92f2fa2110132d27a63c88e8aa995f0209" gracePeriod=15
Sep 10 15:45:55.673831 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.673802    2564 generic.go:356] "Generic (PLEG): container finished" podID="e280aab7-7513-492f-96c3-7361f6058a99" containerID="9a2cd9f3c7ef6434c030dbc566482b92f2fa2110132d27a63c88e8aa995f0209" exitCode=0
Sep 10 15:45:55.674016 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.673869    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66bc9588f5-k64z2" event={"ID":"e280aab7-7513-492f-96c3-7361f6058a99","Type":"ContainerDied","Data":"9a2cd9f3c7ef6434c030dbc566482b92f2fa2110132d27a63c88e8aa995f0209"}
Sep 10 15:45:55.674016 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.673917    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66bc9588f5-k64z2" event={"ID":"e280aab7-7513-492f-96c3-7361f6058a99","Type":"ContainerDied","Data":"4377f99fb31645863152929d141a350ac7a8f3438630d2efa80b93d8b7520435"}
Sep 10 15:45:55.674016 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.673931    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4377f99fb31645863152929d141a350ac7a8f3438630d2efa80b93d8b7520435"
Sep 10 15:45:55.681197 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.681151    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:55.757119 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.757023    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-oauth-serving-cert\") pod \"e280aab7-7513-492f-96c3-7361f6058a99\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") "
Sep 10 15:45:55.757119 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.757087    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-console-config\") pod \"e280aab7-7513-492f-96c3-7361f6058a99\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") "
Sep 10 15:45:55.757119 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.757115    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e280aab7-7513-492f-96c3-7361f6058a99-console-oauth-config\") pod \"e280aab7-7513-492f-96c3-7361f6058a99\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") "
Sep 10 15:45:55.757417 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.757182    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4psb\" (UniqueName: \"kubernetes.io/projected/e280aab7-7513-492f-96c3-7361f6058a99-kube-api-access-x4psb\") pod \"e280aab7-7513-492f-96c3-7361f6058a99\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") "
Sep 10 15:45:55.757417 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.757224    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-service-ca\") pod \"e280aab7-7513-492f-96c3-7361f6058a99\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") "
Sep 10 15:45:55.757417 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.757272    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e280aab7-7513-492f-96c3-7361f6058a99-console-serving-cert\") pod \"e280aab7-7513-492f-96c3-7361f6058a99\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") "
Sep 10 15:45:55.757417 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.757298    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-trusted-ca-bundle\") pod \"e280aab7-7513-492f-96c3-7361f6058a99\" (UID: \"e280aab7-7513-492f-96c3-7361f6058a99\") "
Sep 10 15:45:55.757620 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.757480    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-console-config" (OuterVolumeSpecName: "console-config") pod "e280aab7-7513-492f-96c3-7361f6058a99" (UID: "e280aab7-7513-492f-96c3-7361f6058a99"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:45:55.757620 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.757583    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-console-config\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:45:55.757620 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.757582    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e280aab7-7513-492f-96c3-7361f6058a99" (UID: "e280aab7-7513-492f-96c3-7361f6058a99"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:45:55.757775 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.757648    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-service-ca" (OuterVolumeSpecName: "service-ca") pod "e280aab7-7513-492f-96c3-7361f6058a99" (UID: "e280aab7-7513-492f-96c3-7361f6058a99"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:45:55.757861 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.757841    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e280aab7-7513-492f-96c3-7361f6058a99" (UID: "e280aab7-7513-492f-96c3-7361f6058a99"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:45:55.759618 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.759582    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e280aab7-7513-492f-96c3-7361f6058a99-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e280aab7-7513-492f-96c3-7361f6058a99" (UID: "e280aab7-7513-492f-96c3-7361f6058a99"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:45:55.759618 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.759586    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e280aab7-7513-492f-96c3-7361f6058a99-kube-api-access-x4psb" (OuterVolumeSpecName: "kube-api-access-x4psb") pod "e280aab7-7513-492f-96c3-7361f6058a99" (UID: "e280aab7-7513-492f-96c3-7361f6058a99"). InnerVolumeSpecName "kube-api-access-x4psb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:45:55.759772 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.759592    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e280aab7-7513-492f-96c3-7361f6058a99-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e280aab7-7513-492f-96c3-7361f6058a99" (UID: "e280aab7-7513-492f-96c3-7361f6058a99"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:45:55.858794 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.858755    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:45:55.858794 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.858786    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e280aab7-7513-492f-96c3-7361f6058a99-console-serving-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:45:55.858794 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.858796    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-trusted-ca-bundle\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:45:55.858794 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.858807    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e280aab7-7513-492f-96c3-7361f6058a99-oauth-serving-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:45:55.859053 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.858816    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e280aab7-7513-492f-96c3-7361f6058a99-console-oauth-config\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:45:55.859053 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:55.858824    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4psb\" (UniqueName: \"kubernetes.io/projected/e280aab7-7513-492f-96c3-7361f6058a99-kube-api-access-x4psb\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:45:56.678178 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:56.678139    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66bc9588f5-k64z2"
Sep 10 15:45:56.700059 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:56.700023    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66bc9588f5-k64z2"]
Sep 10 15:45:56.706710 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:56.706682    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66bc9588f5-k64z2"]
Sep 10 15:45:57.671268 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:45:57.671234    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e280aab7-7513-492f-96c3-7361f6058a99" path="/var/lib/kubelet/pods/e280aab7-7513-492f-96c3-7361f6058a99/volumes"
Sep 10 15:46:06.692422 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:06.692368    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bcb99f57d-cn2gt" podUID="26757481-2418-47f1-b199-a2e6e4aa6592" containerName="console" containerID="cri-o://132331f42d6d958b1fe83493ec6c8b34ae594251dfee038264bd520443c8c55b" gracePeriod=15
Sep 10 15:46:06.924857 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:06.924833    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:46:07.052771 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.052674    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq2fk\" (UniqueName: \"kubernetes.io/projected/26757481-2418-47f1-b199-a2e6e4aa6592-kube-api-access-tq2fk\") pod \"26757481-2418-47f1-b199-a2e6e4aa6592\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") "
Sep 10 15:46:07.052771 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.052720    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26757481-2418-47f1-b199-a2e6e4aa6592-console-serving-cert\") pod \"26757481-2418-47f1-b199-a2e6e4aa6592\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") "
Sep 10 15:46:07.052771 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.052746    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-service-ca\") pod \"26757481-2418-47f1-b199-a2e6e4aa6592\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") "
Sep 10 15:46:07.053019 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.052777    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-console-config\") pod \"26757481-2418-47f1-b199-a2e6e4aa6592\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") "
Sep 10 15:46:07.053019 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.052988    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26757481-2418-47f1-b199-a2e6e4aa6592-console-oauth-config\") pod \"26757481-2418-47f1-b199-a2e6e4aa6592\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") "
Sep 10 15:46:07.053120 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.053071    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-oauth-serving-cert\") pod \"26757481-2418-47f1-b199-a2e6e4aa6592\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") "
Sep 10 15:46:07.053267 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.053118    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-trusted-ca-bundle\") pod \"26757481-2418-47f1-b199-a2e6e4aa6592\" (UID: \"26757481-2418-47f1-b199-a2e6e4aa6592\") "
Sep 10 15:46:07.053267 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.053190    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-service-ca" (OuterVolumeSpecName: "service-ca") pod "26757481-2418-47f1-b199-a2e6e4aa6592" (UID: "26757481-2418-47f1-b199-a2e6e4aa6592"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:46:07.053267 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.053203    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-console-config" (OuterVolumeSpecName: "console-config") pod "26757481-2418-47f1-b199-a2e6e4aa6592" (UID: "26757481-2418-47f1-b199-a2e6e4aa6592"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:46:07.053474 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.053457    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:46:07.053544 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.053480    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-console-config\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:46:07.053584 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.053474    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "26757481-2418-47f1-b199-a2e6e4aa6592" (UID: "26757481-2418-47f1-b199-a2e6e4aa6592"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:46:07.053823 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.053797    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "26757481-2418-47f1-b199-a2e6e4aa6592" (UID: "26757481-2418-47f1-b199-a2e6e4aa6592"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:46:07.055127 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.055098    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26757481-2418-47f1-b199-a2e6e4aa6592-kube-api-access-tq2fk" (OuterVolumeSpecName: "kube-api-access-tq2fk") pod "26757481-2418-47f1-b199-a2e6e4aa6592" (UID: "26757481-2418-47f1-b199-a2e6e4aa6592"). InnerVolumeSpecName "kube-api-access-tq2fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:46:07.055127 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.055116    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26757481-2418-47f1-b199-a2e6e4aa6592-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "26757481-2418-47f1-b199-a2e6e4aa6592" (UID: "26757481-2418-47f1-b199-a2e6e4aa6592"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:46:07.055316 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.055196    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26757481-2418-47f1-b199-a2e6e4aa6592-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "26757481-2418-47f1-b199-a2e6e4aa6592" (UID: "26757481-2418-47f1-b199-a2e6e4aa6592"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:46:07.154299 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.154259    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq2fk\" (UniqueName: \"kubernetes.io/projected/26757481-2418-47f1-b199-a2e6e4aa6592-kube-api-access-tq2fk\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:46:07.154299 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.154291    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26757481-2418-47f1-b199-a2e6e4aa6592-console-serving-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:46:07.154299 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.154302    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26757481-2418-47f1-b199-a2e6e4aa6592-console-oauth-config\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:46:07.154299 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.154310    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-oauth-serving-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:46:07.154548 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.154320    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26757481-2418-47f1-b199-a2e6e4aa6592-trusted-ca-bundle\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:46:07.721382 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.721343    2564 generic.go:356] "Generic (PLEG): container finished" podID="26757481-2418-47f1-b199-a2e6e4aa6592" containerID="132331f42d6d958b1fe83493ec6c8b34ae594251dfee038264bd520443c8c55b" exitCode=0
Sep 10 15:46:07.721809 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.721400    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcb99f57d-cn2gt"
Sep 10 15:46:07.721809 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.721434    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcb99f57d-cn2gt" event={"ID":"26757481-2418-47f1-b199-a2e6e4aa6592","Type":"ContainerDied","Data":"132331f42d6d958b1fe83493ec6c8b34ae594251dfee038264bd520443c8c55b"}
Sep 10 15:46:07.721809 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.721488    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcb99f57d-cn2gt" event={"ID":"26757481-2418-47f1-b199-a2e6e4aa6592","Type":"ContainerDied","Data":"90057b04d5845de6c76163008cb33741caf057155c98a20e3cd65e07328dd94d"}
Sep 10 15:46:07.721809 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.721508    2564 scope.go:117] "RemoveContainer" containerID="132331f42d6d958b1fe83493ec6c8b34ae594251dfee038264bd520443c8c55b"
Sep 10 15:46:07.729748 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.729732    2564 scope.go:117] "RemoveContainer" containerID="132331f42d6d958b1fe83493ec6c8b34ae594251dfee038264bd520443c8c55b"
Sep 10 15:46:07.730014 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:46:07.729995    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"132331f42d6d958b1fe83493ec6c8b34ae594251dfee038264bd520443c8c55b\": container with ID starting with 132331f42d6d958b1fe83493ec6c8b34ae594251dfee038264bd520443c8c55b not found: ID does not exist" containerID="132331f42d6d958b1fe83493ec6c8b34ae594251dfee038264bd520443c8c55b"
Sep 10 15:46:07.730081 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.730024    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132331f42d6d958b1fe83493ec6c8b34ae594251dfee038264bd520443c8c55b"} err="failed to get container status \"132331f42d6d958b1fe83493ec6c8b34ae594251dfee038264bd520443c8c55b\": rpc error: code = NotFound desc = could not find container \"132331f42d6d958b1fe83493ec6c8b34ae594251dfee038264bd520443c8c55b\": container with ID starting with 132331f42d6d958b1fe83493ec6c8b34ae594251dfee038264bd520443c8c55b not found: ID does not exist"
Sep 10 15:46:07.738817 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.738791    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bcb99f57d-cn2gt"]
Sep 10 15:46:07.743026 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:07.743001    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bcb99f57d-cn2gt"]
Sep 10 15:46:09.671433 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:09.671394    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26757481-2418-47f1-b199-a2e6e4aa6592" path="/var/lib/kubelet/pods/26757481-2418-47f1-b199-a2e6e4aa6592/volumes"
Sep 10 15:46:58.695991 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:58.695951    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kh6zg"]
Sep 10 15:46:58.696545 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:58.696408    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26757481-2418-47f1-b199-a2e6e4aa6592" containerName="console"
Sep 10 15:46:58.696545 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:58.696425    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="26757481-2418-47f1-b199-a2e6e4aa6592" containerName="console"
Sep 10 15:46:58.696545 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:58.696456    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e280aab7-7513-492f-96c3-7361f6058a99" containerName="console"
Sep 10 15:46:58.696545 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:58.696464    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e280aab7-7513-492f-96c3-7361f6058a99" containerName="console"
Sep 10 15:46:58.696545 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:58.696543    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="26757481-2418-47f1-b199-a2e6e4aa6592" containerName="console"
Sep 10 15:46:58.696798 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:58.696560    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e280aab7-7513-492f-96c3-7361f6058a99" containerName="console"
Sep 10 15:46:58.698424 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:58.698404    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-kh6zg"
Sep 10 15:46:58.700883 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:58.700859    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 15:46:58.701834 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:58.701818    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-np5mh\""
Sep 10 15:46:58.701898 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:58.701842    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 15:46:58.708720 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:58.708698    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kh6zg"]
Sep 10 15:46:58.769915 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:58.769881    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdvqb\" (UniqueName: \"kubernetes.io/projected/f41486e5-bba8-4b1d-874b-bb7d305d7a1f-kube-api-access-qdvqb\") pod \"cert-manager-55b4d954d5-kh6zg\" (UID: \"f41486e5-bba8-4b1d-874b-bb7d305d7a1f\") " pod="cert-manager/cert-manager-55b4d954d5-kh6zg"
Sep 10 15:46:58.770087 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:58.769928    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f41486e5-bba8-4b1d-874b-bb7d305d7a1f-bound-sa-token\") pod \"cert-manager-55b4d954d5-kh6zg\" (UID: \"f41486e5-bba8-4b1d-874b-bb7d305d7a1f\") " pod="cert-manager/cert-manager-55b4d954d5-kh6zg"
Sep 10 15:46:58.871033 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:58.870999    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdvqb\" (UniqueName: \"kubernetes.io/projected/f41486e5-bba8-4b1d-874b-bb7d305d7a1f-kube-api-access-qdvqb\") pod \"cert-manager-55b4d954d5-kh6zg\" (UID: \"f41486e5-bba8-4b1d-874b-bb7d305d7a1f\") " pod="cert-manager/cert-manager-55b4d954d5-kh6zg"
Sep 10 15:46:58.871287 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:58.871048    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f41486e5-bba8-4b1d-874b-bb7d305d7a1f-bound-sa-token\") pod \"cert-manager-55b4d954d5-kh6zg\" (UID: \"f41486e5-bba8-4b1d-874b-bb7d305d7a1f\") " pod="cert-manager/cert-manager-55b4d954d5-kh6zg"
Sep 10 15:46:58.879910 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:58.879879    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f41486e5-bba8-4b1d-874b-bb7d305d7a1f-bound-sa-token\") pod \"cert-manager-55b4d954d5-kh6zg\" (UID: \"f41486e5-bba8-4b1d-874b-bb7d305d7a1f\") " pod="cert-manager/cert-manager-55b4d954d5-kh6zg"
Sep 10 15:46:58.880015 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:58.879984    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdvqb\" (UniqueName: \"kubernetes.io/projected/f41486e5-bba8-4b1d-874b-bb7d305d7a1f-kube-api-access-qdvqb\") pod \"cert-manager-55b4d954d5-kh6zg\" (UID: \"f41486e5-bba8-4b1d-874b-bb7d305d7a1f\") " pod="cert-manager/cert-manager-55b4d954d5-kh6zg"
Sep 10 15:46:59.014653 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:59.014568    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-kh6zg"
Sep 10 15:46:59.284219 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:59.284092    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kh6zg"]
Sep 10 15:46:59.286866 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:46:59.286833    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf41486e5_bba8_4b1d_874b_bb7d305d7a1f.slice/crio-f14092c85d3aea9b1ea5e907301fa8bc0ede981ca922c42de8d980ff61fc3fe0 WatchSource:0}: Error finding container f14092c85d3aea9b1ea5e907301fa8bc0ede981ca922c42de8d980ff61fc3fe0: Status 404 returned error can't find the container with id f14092c85d3aea9b1ea5e907301fa8bc0ede981ca922c42de8d980ff61fc3fe0
Sep 10 15:46:59.931523 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:46:59.931486    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-kh6zg" event={"ID":"f41486e5-bba8-4b1d-874b-bb7d305d7a1f","Type":"ContainerStarted","Data":"f14092c85d3aea9b1ea5e907301fa8bc0ede981ca922c42de8d980ff61fc3fe0"}
Sep 10 15:47:02.261663 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:02.261626    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-snm4z_f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f/console-operator/1.log"
Sep 10 15:47:02.262116 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:02.261936    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-snm4z_f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f/console-operator/1.log"
Sep 10 15:47:02.272564 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:02.272543    2564 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 15:47:02.946635 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:02.946605    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-kh6zg" event={"ID":"f41486e5-bba8-4b1d-874b-bb7d305d7a1f","Type":"ContainerStarted","Data":"69af14a2696b56e303949db6abff77d981365f4821700e5ad28e9d4eb628b2e4"}
Sep 10 15:47:02.963490 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:02.963439    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-kh6zg" podStartSLOduration=1.968611921 podStartE2EDuration="4.963424611s" totalImagesPullingTime="2.99481269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:02.962434715 +0000 UTC m=+301.766753979" watchObservedRunningTime="2026-09-10 15:47:02.963424611 +0000 UTC m=+301.767743871"
Sep 10 15:47:19.755234 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.755195    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"]
Sep 10 15:47:19.758099 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.758077    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"
Sep 10 15:47:19.760674 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.760638    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 15:47:19.760976 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.760959    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-s4hf4\""
Sep 10 15:47:19.761328 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.761313    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 15:47:19.761382 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.761346    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 15:47:19.761439 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.761318    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 15:47:19.780236 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.780211    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"]
Sep 10 15:47:19.846418 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.846379    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a31469b0-b6aa-4281-a68a-54961f930bfd-webhook-cert\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-5vz94\" (UID: \"a31469b0-b6aa-4281-a68a-54961f930bfd\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"
Sep 10 15:47:19.846596 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.846427    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7nmn\" (UniqueName: \"kubernetes.io/projected/a31469b0-b6aa-4281-a68a-54961f930bfd-kube-api-access-n7nmn\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-5vz94\" (UID: \"a31469b0-b6aa-4281-a68a-54961f930bfd\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"
Sep 10 15:47:19.846596 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.846496    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a31469b0-b6aa-4281-a68a-54961f930bfd-manifests\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-5vz94\" (UID: \"a31469b0-b6aa-4281-a68a-54961f930bfd\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"
Sep 10 15:47:19.846596 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.846535    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a31469b0-b6aa-4281-a68a-54961f930bfd-tmp\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-5vz94\" (UID: \"a31469b0-b6aa-4281-a68a-54961f930bfd\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"
Sep 10 15:47:19.846596 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.846555    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a31469b0-b6aa-4281-a68a-54961f930bfd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-5vz94\" (UID: \"a31469b0-b6aa-4281-a68a-54961f930bfd\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"
Sep 10 15:47:19.947851 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.947799    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a31469b0-b6aa-4281-a68a-54961f930bfd-manifests\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-5vz94\" (UID: \"a31469b0-b6aa-4281-a68a-54961f930bfd\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"
Sep 10 15:47:19.948049 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.947889    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a31469b0-b6aa-4281-a68a-54961f930bfd-tmp\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-5vz94\" (UID: \"a31469b0-b6aa-4281-a68a-54961f930bfd\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"
Sep 10 15:47:19.948049 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.947919    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a31469b0-b6aa-4281-a68a-54961f930bfd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-5vz94\" (UID: \"a31469b0-b6aa-4281-a68a-54961f930bfd\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"
Sep 10 15:47:19.948049 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.947984    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a31469b0-b6aa-4281-a68a-54961f930bfd-webhook-cert\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-5vz94\" (UID: \"a31469b0-b6aa-4281-a68a-54961f930bfd\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"
Sep 10 15:47:19.948049 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.948015    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7nmn\" (UniqueName: \"kubernetes.io/projected/a31469b0-b6aa-4281-a68a-54961f930bfd-kube-api-access-n7nmn\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-5vz94\" (UID: \"a31469b0-b6aa-4281-a68a-54961f930bfd\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"
Sep 10 15:47:19.948343 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.948323    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a31469b0-b6aa-4281-a68a-54961f930bfd-manifests\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-5vz94\" (UID: \"a31469b0-b6aa-4281-a68a-54961f930bfd\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"
Sep 10 15:47:19.950380 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.950359    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a31469b0-b6aa-4281-a68a-54961f930bfd-tmp\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-5vz94\" (UID: \"a31469b0-b6aa-4281-a68a-54961f930bfd\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"
Sep 10 15:47:19.950580 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.950564    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a31469b0-b6aa-4281-a68a-54961f930bfd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-5vz94\" (UID: \"a31469b0-b6aa-4281-a68a-54961f930bfd\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"
Sep 10 15:47:19.950625 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.950581    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a31469b0-b6aa-4281-a68a-54961f930bfd-webhook-cert\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-5vz94\" (UID: \"a31469b0-b6aa-4281-a68a-54961f930bfd\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"
Sep 10 15:47:19.956328 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:19.956305    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7nmn\" (UniqueName: \"kubernetes.io/projected/a31469b0-b6aa-4281-a68a-54961f930bfd-kube-api-access-n7nmn\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-5vz94\" (UID: \"a31469b0-b6aa-4281-a68a-54961f930bfd\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"
Sep 10 15:47:20.068716 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:20.068622    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"
Sep 10 15:47:20.357117 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:20.357053    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"]
Sep 10 15:47:20.359585 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:47:20.359559    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31469b0_b6aa_4281_a68a_54961f930bfd.slice/crio-b0f7a2eb3becbb427a58e78136db34f0ca75a9ff03d8e7ab7874ab3574e8cdfe WatchSource:0}: Error finding container b0f7a2eb3becbb427a58e78136db34f0ca75a9ff03d8e7ab7874ab3574e8cdfe: Status 404 returned error can't find the container with id b0f7a2eb3becbb427a58e78136db34f0ca75a9ff03d8e7ab7874ab3574e8cdfe
Sep 10 15:47:20.361507 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:20.361490    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:47:21.022302 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:21.022267    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94" event={"ID":"a31469b0-b6aa-4281-a68a-54961f930bfd","Type":"ContainerStarted","Data":"b0f7a2eb3becbb427a58e78136db34f0ca75a9ff03d8e7ab7874ab3574e8cdfe"}
Sep 10 15:47:24.037636 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:24.037593    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94" event={"ID":"a31469b0-b6aa-4281-a68a-54961f930bfd","Type":"ContainerStarted","Data":"87ace3e2766f631b69566e441336bdade99467510e3f8e4ebaa6019e1e717ccf"}
Sep 10 15:47:25.043227 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:25.043136    2564 generic.go:356] "Generic (PLEG): container finished" podID="a31469b0-b6aa-4281-a68a-54961f930bfd" containerID="87ace3e2766f631b69566e441336bdade99467510e3f8e4ebaa6019e1e717ccf" exitCode=0
Sep 10 15:47:25.043227 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:25.043195    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94" event={"ID":"a31469b0-b6aa-4281-a68a-54961f930bfd","Type":"ContainerDied","Data":"87ace3e2766f631b69566e441336bdade99467510e3f8e4ebaa6019e1e717ccf"}
Sep 10 15:47:26.050076 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:26.050037    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94" event={"ID":"a31469b0-b6aa-4281-a68a-54961f930bfd","Type":"ContainerStarted","Data":"3918adb6927625f5b0c238fcb333aecf599b9304c12d517b25e38e1e65096963"}
Sep 10 15:47:26.050496 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:26.050205    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"
Sep 10 15:47:26.076398 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:26.076350    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94" podStartSLOduration=2.446064323 podStartE2EDuration="7.076333032s" totalImagesPullingTime="3.57914158s" totalInitContainerRuntime="1.051127129s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:26.074343846 +0000 UTC m=+324.878663109" watchObservedRunningTime="2026-09-10 15:47:26.076333032 +0000 UTC m=+324.880652293"
Sep 10 15:47:37.057645 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:37.057609    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-5vz94"
Sep 10 15:47:39.094914 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:39.094878    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6896b79569-vjk5p"]
Sep 10 15:47:39.107065 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:39.107042    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6896b79569-vjk5p"
Sep 10 15:47:39.110050 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:39.109822    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 15:47:39.110050 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:39.109860    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 15:47:39.110050 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:39.109897    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vjsdp\""
Sep 10 15:47:39.110050 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:39.109945    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 15:47:39.110050 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:39.109901    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 15:47:39.113020 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:39.112995    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6896b79569-vjk5p"]
Sep 10 15:47:39.213635 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:39.213592    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bzkk\" (UniqueName: \"kubernetes.io/projected/5056caa3-d566-456b-935b-853c5b3be141-kube-api-access-7bzkk\") pod \"kube-auth-proxy-6896b79569-vjk5p\" (UID: \"5056caa3-d566-456b-935b-853c5b3be141\") " pod="openshift-ingress/kube-auth-proxy-6896b79569-vjk5p"
Sep 10 15:47:39.213818 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:39.213699    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5056caa3-d566-456b-935b-853c5b3be141-tmp\") pod \"kube-auth-proxy-6896b79569-vjk5p\" (UID: \"5056caa3-d566-456b-935b-853c5b3be141\") " pod="openshift-ingress/kube-auth-proxy-6896b79569-vjk5p"
Sep 10 15:47:39.213818 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:39.213728    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5056caa3-d566-456b-935b-853c5b3be141-tls-certs\") pod \"kube-auth-proxy-6896b79569-vjk5p\" (UID: \"5056caa3-d566-456b-935b-853c5b3be141\") " pod="openshift-ingress/kube-auth-proxy-6896b79569-vjk5p"
Sep 10 15:47:39.314458 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:39.314427    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bzkk\" (UniqueName: \"kubernetes.io/projected/5056caa3-d566-456b-935b-853c5b3be141-kube-api-access-7bzkk\") pod \"kube-auth-proxy-6896b79569-vjk5p\" (UID: \"5056caa3-d566-456b-935b-853c5b3be141\") " pod="openshift-ingress/kube-auth-proxy-6896b79569-vjk5p"
Sep 10 15:47:39.314648 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:39.314482    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5056caa3-d566-456b-935b-853c5b3be141-tmp\") pod \"kube-auth-proxy-6896b79569-vjk5p\" (UID: \"5056caa3-d566-456b-935b-853c5b3be141\") " pod="openshift-ingress/kube-auth-proxy-6896b79569-vjk5p"
Sep 10 15:47:39.314648 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:39.314507    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5056caa3-d566-456b-935b-853c5b3be141-tls-certs\") pod \"kube-auth-proxy-6896b79569-vjk5p\" (UID: \"5056caa3-d566-456b-935b-853c5b3be141\") " pod="openshift-ingress/kube-auth-proxy-6896b79569-vjk5p"
Sep 10 15:47:39.314648 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:47:39.314598    2564 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 10 15:47:39.314813 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:47:39.314660    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5056caa3-d566-456b-935b-853c5b3be141-tls-certs podName:5056caa3-d566-456b-935b-853c5b3be141 nodeName:}" failed. No retries permitted until 2026-09-10 15:47:39.814644376 +0000 UTC m=+338.618963615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5056caa3-d566-456b-935b-853c5b3be141-tls-certs") pod "kube-auth-proxy-6896b79569-vjk5p" (UID: "5056caa3-d566-456b-935b-853c5b3be141") : secret "kube-auth-proxy-tls" not found
Sep 10 15:47:39.316902 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:39.316882    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5056caa3-d566-456b-935b-853c5b3be141-tmp\") pod \"kube-auth-proxy-6896b79569-vjk5p\" (UID: \"5056caa3-d566-456b-935b-853c5b3be141\") " pod="openshift-ingress/kube-auth-proxy-6896b79569-vjk5p"
Sep 10 15:47:39.323691 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:39.323666    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bzkk\" (UniqueName: \"kubernetes.io/projected/5056caa3-d566-456b-935b-853c5b3be141-kube-api-access-7bzkk\") pod \"kube-auth-proxy-6896b79569-vjk5p\" (UID: \"5056caa3-d566-456b-935b-853c5b3be141\") " pod="openshift-ingress/kube-auth-proxy-6896b79569-vjk5p"
Sep 10 15:47:39.818913 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:39.818857    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5056caa3-d566-456b-935b-853c5b3be141-tls-certs\") pod \"kube-auth-proxy-6896b79569-vjk5p\" (UID: \"5056caa3-d566-456b-935b-853c5b3be141\") " pod="openshift-ingress/kube-auth-proxy-6896b79569-vjk5p"
Sep 10 15:47:39.821503 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:39.821481    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5056caa3-d566-456b-935b-853c5b3be141-tls-certs\") pod \"kube-auth-proxy-6896b79569-vjk5p\" (UID: \"5056caa3-d566-456b-935b-853c5b3be141\") " pod="openshift-ingress/kube-auth-proxy-6896b79569-vjk5p"
Sep 10 15:47:40.018147 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.018108    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6896b79569-vjk5p"
Sep 10 15:47:40.290466 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.290441    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6896b79569-vjk5p"]
Sep 10 15:47:40.292923 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:47:40.292895    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5056caa3_d566_456b_935b_853c5b3be141.slice/crio-d5de19751fab86787319c7d0cc5379cabeb7247f5fbf0f0b245433798765250e WatchSource:0}: Error finding container d5de19751fab86787319c7d0cc5379cabeb7247f5fbf0f0b245433798765250e: Status 404 returned error can't find the container with id d5de19751fab86787319c7d0cc5379cabeb7247f5fbf0f0b245433798765250e
Sep 10 15:47:40.351545 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.351464    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"]
Sep 10 15:47:40.354370 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.354347    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.356708 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.356686    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 10 15:47:40.356842 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.356752    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 15:47:40.356842 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.356772    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zfrwz\""
Sep 10 15:47:40.356842 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.356686    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 10 15:47:40.356989 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.356689    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 10 15:47:40.368419 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.368399    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"]
Sep 10 15:47:40.524316 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.524278    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xmvc7\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.524316 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.524332    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xmvc7\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.524574 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.524404    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xmvc7\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.524574 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.524492    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e02ee6-5f5a-4568-9508-4ef41b7163d5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xmvc7\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.524574 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.524527    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26e02ee6-5f5a-4568-9508-4ef41b7163d5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xmvc7\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.524574 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.524544    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xmvc7\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.524730 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.524619    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf6hh\" (UniqueName: \"kubernetes.io/projected/26e02ee6-5f5a-4568-9508-4ef41b7163d5-kube-api-access-cf6hh\") pod \"istiod-openshift-gateway-7b7995d5cd-xmvc7\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.625789 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.625714    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf6hh\" (UniqueName: \"kubernetes.io/projected/26e02ee6-5f5a-4568-9508-4ef41b7163d5-kube-api-access-cf6hh\") pod \"istiod-openshift-gateway-7b7995d5cd-xmvc7\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.625895 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.625798    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xmvc7\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.625895 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.625829    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xmvc7\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.625895 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.625868    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xmvc7\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.625998 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.625903    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e02ee6-5f5a-4568-9508-4ef41b7163d5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xmvc7\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.625998 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.625928    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26e02ee6-5f5a-4568-9508-4ef41b7163d5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xmvc7\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.625998 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.625954    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xmvc7\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.626592 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.626564    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xmvc7\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.628502 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.628478    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26e02ee6-5f5a-4568-9508-4ef41b7163d5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xmvc7\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.628587 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.628504    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xmvc7\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.628668 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.628651    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e02ee6-5f5a-4568-9508-4ef41b7163d5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xmvc7\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.628794 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.628776    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xmvc7\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.635706 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.635677    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xmvc7\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.635932 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.635914    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf6hh\" (UniqueName: \"kubernetes.io/projected/26e02ee6-5f5a-4568-9508-4ef41b7163d5-kube-api-access-cf6hh\") pod \"istiod-openshift-gateway-7b7995d5cd-xmvc7\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.664395 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.664366    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:40.939584 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:40.939559    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"]
Sep 10 15:47:40.942139 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:47:40.942115    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e02ee6_5f5a_4568_9508_4ef41b7163d5.slice/crio-173c8bef46a0c85a9cd9bdf50bf773afdacafe18afbb3b8e5935c9b1160a7be7 WatchSource:0}: Error finding container 173c8bef46a0c85a9cd9bdf50bf773afdacafe18afbb3b8e5935c9b1160a7be7: Status 404 returned error can't find the container with id 173c8bef46a0c85a9cd9bdf50bf773afdacafe18afbb3b8e5935c9b1160a7be7
Sep 10 15:47:41.116235 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:41.116200    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7" event={"ID":"26e02ee6-5f5a-4568-9508-4ef41b7163d5","Type":"ContainerStarted","Data":"173c8bef46a0c85a9cd9bdf50bf773afdacafe18afbb3b8e5935c9b1160a7be7"}
Sep 10 15:47:41.117364 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:41.117329    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6896b79569-vjk5p" event={"ID":"5056caa3-d566-456b-935b-853c5b3be141","Type":"ContainerStarted","Data":"d5de19751fab86787319c7d0cc5379cabeb7247f5fbf0f0b245433798765250e"}
Sep 10 15:47:44.659295 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:44.659252    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:47:44.659640 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:44.659328    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:47:45.139687 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:45.139647    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7" event={"ID":"26e02ee6-5f5a-4568-9508-4ef41b7163d5","Type":"ContainerStarted","Data":"d6baafdf7c2f91b18b05ebc5a1deec46126df5343c1cbe4a1d0d631275df4fcd"}
Sep 10 15:47:45.139902 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:45.139822    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:45.141576 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:45.141546    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6896b79569-vjk5p" event={"ID":"5056caa3-d566-456b-935b-853c5b3be141","Type":"ContainerStarted","Data":"2d3dc0027518c43db89ebded8e78d88a64409ff56dd596075c21b8310bcfb564"}
Sep 10 15:47:45.142057 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:45.142030    2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-xmvc7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 15:47:45.142205 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:45.142089    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7" podUID="26e02ee6-5f5a-4568-9508-4ef41b7163d5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 15:47:45.160527 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:45.160468    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7" podStartSLOduration=1.445839771 podStartE2EDuration="5.160452165s" totalImagesPullingTime="3.714612394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:45.160063638 +0000 UTC m=+343.964382900" watchObservedRunningTime="2026-09-10 15:47:45.160452165 +0000 UTC m=+343.964771425"
Sep 10 15:47:45.176047 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:45.175989    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6896b79569-vjk5p" podStartSLOduration=1.8099624479999998 podStartE2EDuration="6.17597334s" totalImagesPullingTime="4.366010892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:45.175444361 +0000 UTC m=+343.979763636" watchObservedRunningTime="2026-09-10 15:47:45.17597334 +0000 UTC m=+343.980292600"
Sep 10 15:47:46.147031 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:46.146997    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:47:50.522787 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:50.522754    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2rkvm"]
Sep 10 15:47:50.526439 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:50.526421    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2rkvm"
Sep 10 15:47:50.528664 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:50.528643    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 10 15:47:50.538872 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:50.538843    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2rkvm"]
Sep 10 15:47:50.613583 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:50.613543    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kglc\" (UniqueName: \"kubernetes.io/projected/f0e4d03d-ca8f-4f66-9a0f-5e1885b280f1-kube-api-access-8kglc\") pod \"model-serving-api-54d89bd5fc-2rkvm\" (UID: \"f0e4d03d-ca8f-4f66-9a0f-5e1885b280f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-2rkvm"
Sep 10 15:47:50.613805 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:50.613610    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e4d03d-ca8f-4f66-9a0f-5e1885b280f1-tls-certs\") pod \"model-serving-api-54d89bd5fc-2rkvm\" (UID: \"f0e4d03d-ca8f-4f66-9a0f-5e1885b280f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-2rkvm"
Sep 10 15:47:50.715261 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:50.715219    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kglc\" (UniqueName: \"kubernetes.io/projected/f0e4d03d-ca8f-4f66-9a0f-5e1885b280f1-kube-api-access-8kglc\") pod \"model-serving-api-54d89bd5fc-2rkvm\" (UID: \"f0e4d03d-ca8f-4f66-9a0f-5e1885b280f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-2rkvm"
Sep 10 15:47:50.715468 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:50.715285    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e4d03d-ca8f-4f66-9a0f-5e1885b280f1-tls-certs\") pod \"model-serving-api-54d89bd5fc-2rkvm\" (UID: \"f0e4d03d-ca8f-4f66-9a0f-5e1885b280f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-2rkvm"
Sep 10 15:47:50.717868 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:50.717842    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e4d03d-ca8f-4f66-9a0f-5e1885b280f1-tls-certs\") pod \"model-serving-api-54d89bd5fc-2rkvm\" (UID: \"f0e4d03d-ca8f-4f66-9a0f-5e1885b280f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-2rkvm"
Sep 10 15:47:50.724514 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:50.724491    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kglc\" (UniqueName: \"kubernetes.io/projected/f0e4d03d-ca8f-4f66-9a0f-5e1885b280f1-kube-api-access-8kglc\") pod \"model-serving-api-54d89bd5fc-2rkvm\" (UID: \"f0e4d03d-ca8f-4f66-9a0f-5e1885b280f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-2rkvm"
Sep 10 15:47:50.836769 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:50.836663    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2rkvm"
Sep 10 15:47:51.110634 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:51.110609    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2rkvm"]
Sep 10 15:47:51.112411 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:47:51.112382    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e4d03d_ca8f_4f66_9a0f_5e1885b280f1.slice/crio-409209de514d52e06baedd72fa73b4754cbaea47926a7635d74f272950f94bec WatchSource:0}: Error finding container 409209de514d52e06baedd72fa73b4754cbaea47926a7635d74f272950f94bec: Status 404 returned error can't find the container with id 409209de514d52e06baedd72fa73b4754cbaea47926a7635d74f272950f94bec
Sep 10 15:47:51.169011 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:51.168984    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2rkvm" event={"ID":"f0e4d03d-ca8f-4f66-9a0f-5e1885b280f1","Type":"ContainerStarted","Data":"409209de514d52e06baedd72fa73b4754cbaea47926a7635d74f272950f94bec"}
Sep 10 15:47:53.182019 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:53.180504    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2rkvm" event={"ID":"f0e4d03d-ca8f-4f66-9a0f-5e1885b280f1","Type":"ContainerStarted","Data":"c463aef2468d067cb51788cc0a0296f9fb2711c5438addd596b254207aaabe33"}
Sep 10 15:47:53.182019 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:53.181392    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-2rkvm"
Sep 10 15:47:53.200981 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:53.200927    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-2rkvm" podStartSLOduration=1.382292471 podStartE2EDuration="3.200911506s" totalImagesPullingTime="1.818619035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:53.19944048 +0000 UTC m=+352.003759828" watchObservedRunningTime="2026-09-10 15:47:53.200911506 +0000 UTC m=+352.005230767"
Sep 10 15:47:59.230724 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.230694    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mb7mt"]
Sep 10 15:47:59.233992 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.233977    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mb7mt"
Sep 10 15:47:59.236540 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.236513    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 10 15:47:59.236867 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.236850    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-f7fqn\""
Sep 10 15:47:59.255472 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.255444    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mb7mt"]
Sep 10 15:47:59.256905 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.256877    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vsdlt"]
Sep 10 15:47:59.260280 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.260257    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vsdlt"
Sep 10 15:47:59.263032 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.263009    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-p6xlp\""
Sep 10 15:47:59.263224 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.263196    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 10 15:47:59.289241 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.289207    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vsdlt"]
Sep 10 15:47:59.293724 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.293525    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nkfq\" (UniqueName: \"kubernetes.io/projected/69c36949-a1a5-4c1b-a250-4a08cb639da3-kube-api-access-5nkfq\") pod \"kserve-controller-manager-d6f4b65b8-mb7mt\" (UID: \"69c36949-a1a5-4c1b-a250-4a08cb639da3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mb7mt"
Sep 10 15:47:59.293724 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.293583    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69c36949-a1a5-4c1b-a250-4a08cb639da3-cert\") pod \"kserve-controller-manager-d6f4b65b8-mb7mt\" (UID: \"69c36949-a1a5-4c1b-a250-4a08cb639da3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mb7mt"
Sep 10 15:47:59.394553 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.394516    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab9b3bee-e44b-4a95-a430-7a712f5c323b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vsdlt\" (UID: \"ab9b3bee-e44b-4a95-a430-7a712f5c323b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vsdlt"
Sep 10 15:47:59.394743 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.394566    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nkfq\" (UniqueName: \"kubernetes.io/projected/69c36949-a1a5-4c1b-a250-4a08cb639da3-kube-api-access-5nkfq\") pod \"kserve-controller-manager-d6f4b65b8-mb7mt\" (UID: \"69c36949-a1a5-4c1b-a250-4a08cb639da3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mb7mt"
Sep 10 15:47:59.394743 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.394595    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69c36949-a1a5-4c1b-a250-4a08cb639da3-cert\") pod \"kserve-controller-manager-d6f4b65b8-mb7mt\" (UID: \"69c36949-a1a5-4c1b-a250-4a08cb639da3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mb7mt"
Sep 10 15:47:59.394743 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.394650    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr2fw\" (UniqueName: \"kubernetes.io/projected/ab9b3bee-e44b-4a95-a430-7a712f5c323b-kube-api-access-qr2fw\") pod \"llmisvc-controller-manager-777c6bd7d5-vsdlt\" (UID: \"ab9b3bee-e44b-4a95-a430-7a712f5c323b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vsdlt"
Sep 10 15:47:59.394860 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:47:59.394785    2564 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 10 15:47:59.394860 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:47:59.394841    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69c36949-a1a5-4c1b-a250-4a08cb639da3-cert podName:69c36949-a1a5-4c1b-a250-4a08cb639da3 nodeName:}" failed. No retries permitted until 2026-09-10 15:47:59.89482663 +0000 UTC m=+358.699145868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/69c36949-a1a5-4c1b-a250-4a08cb639da3-cert") pod "kserve-controller-manager-d6f4b65b8-mb7mt" (UID: "69c36949-a1a5-4c1b-a250-4a08cb639da3") : secret "kserve-webhook-server-cert" not found
Sep 10 15:47:59.409006 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.408976    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nkfq\" (UniqueName: \"kubernetes.io/projected/69c36949-a1a5-4c1b-a250-4a08cb639da3-kube-api-access-5nkfq\") pod \"kserve-controller-manager-d6f4b65b8-mb7mt\" (UID: \"69c36949-a1a5-4c1b-a250-4a08cb639da3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mb7mt"
Sep 10 15:47:59.501329 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.501233    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab9b3bee-e44b-4a95-a430-7a712f5c323b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vsdlt\" (UID: \"ab9b3bee-e44b-4a95-a430-7a712f5c323b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vsdlt"
Sep 10 15:47:59.501503 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.501342    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr2fw\" (UniqueName: \"kubernetes.io/projected/ab9b3bee-e44b-4a95-a430-7a712f5c323b-kube-api-access-qr2fw\") pod \"llmisvc-controller-manager-777c6bd7d5-vsdlt\" (UID: \"ab9b3bee-e44b-4a95-a430-7a712f5c323b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vsdlt"
Sep 10 15:47:59.503911 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.503890    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab9b3bee-e44b-4a95-a430-7a712f5c323b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vsdlt\" (UID: \"ab9b3bee-e44b-4a95-a430-7a712f5c323b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vsdlt"
Sep 10 15:47:59.510711 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.510686    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr2fw\" (UniqueName: \"kubernetes.io/projected/ab9b3bee-e44b-4a95-a430-7a712f5c323b-kube-api-access-qr2fw\") pod \"llmisvc-controller-manager-777c6bd7d5-vsdlt\" (UID: \"ab9b3bee-e44b-4a95-a430-7a712f5c323b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vsdlt"
Sep 10 15:47:59.570010 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.569951    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vsdlt"
Sep 10 15:47:59.857368 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.857343    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vsdlt"]
Sep 10 15:47:59.859374 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:47:59.859351    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podab9b3bee_e44b_4a95_a430_7a712f5c323b.slice/crio-57b3b8a4b99078d030ade54c8649efcb5351df089596dd96b56057bcd38e937e WatchSource:0}: Error finding container 57b3b8a4b99078d030ade54c8649efcb5351df089596dd96b56057bcd38e937e: Status 404 returned error can't find the container with id 57b3b8a4b99078d030ade54c8649efcb5351df089596dd96b56057bcd38e937e
Sep 10 15:47:59.905961 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.905924    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69c36949-a1a5-4c1b-a250-4a08cb639da3-cert\") pod \"kserve-controller-manager-d6f4b65b8-mb7mt\" (UID: \"69c36949-a1a5-4c1b-a250-4a08cb639da3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mb7mt"
Sep 10 15:47:59.908443 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:47:59.908416    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69c36949-a1a5-4c1b-a250-4a08cb639da3-cert\") pod \"kserve-controller-manager-d6f4b65b8-mb7mt\" (UID: \"69c36949-a1a5-4c1b-a250-4a08cb639da3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mb7mt"
Sep 10 15:48:00.144258 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:00.144142    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mb7mt"
Sep 10 15:48:00.215851 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:00.215804    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vsdlt" event={"ID":"ab9b3bee-e44b-4a95-a430-7a712f5c323b","Type":"ContainerStarted","Data":"57b3b8a4b99078d030ade54c8649efcb5351df089596dd96b56057bcd38e937e"}
Sep 10 15:48:00.425885 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:00.425859    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mb7mt"]
Sep 10 15:48:00.428496 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:48:00.428470    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c36949_a1a5_4c1b_a250_4a08cb639da3.slice/crio-b04de46f3c4fc5480226e2affa450eec2c7c0fa73fc4374dd576ced792866238 WatchSource:0}: Error finding container b04de46f3c4fc5480226e2affa450eec2c7c0fa73fc4374dd576ced792866238: Status 404 returned error can't find the container with id b04de46f3c4fc5480226e2affa450eec2c7c0fa73fc4374dd576ced792866238
Sep 10 15:48:01.182393 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.182358    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"]
Sep 10 15:48:01.182637 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.182609    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7" podUID="26e02ee6-5f5a-4568-9508-4ef41b7163d5" containerName="discovery" containerID="cri-o://d6baafdf7c2f91b18b05ebc5a1deec46126df5343c1cbe4a1d0d631275df4fcd" gracePeriod=30
Sep 10 15:48:01.224271 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.224231    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mb7mt" event={"ID":"69c36949-a1a5-4c1b-a250-4a08cb639da3","Type":"ContainerStarted","Data":"b04de46f3c4fc5480226e2affa450eec2c7c0fa73fc4374dd576ced792866238"}
Sep 10 15:48:01.611000 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.610973    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:48:01.722249 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.722218    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-csr-dns-cert\") pod \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") "
Sep 10 15:48:01.722412 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.722282    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e02ee6-5f5a-4568-9508-4ef41b7163d5-local-certs\") pod \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") "
Sep 10 15:48:01.722412 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.722342    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-csr-ca-configmap\") pod \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") "
Sep 10 15:48:01.722412 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.722366    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26e02ee6-5f5a-4568-9508-4ef41b7163d5-cacerts\") pod \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") "
Sep 10 15:48:01.722575 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.722426    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-kubeconfig\") pod \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") "
Sep 10 15:48:01.722575 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.722461    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf6hh\" (UniqueName: \"kubernetes.io/projected/26e02ee6-5f5a-4568-9508-4ef41b7163d5-kube-api-access-cf6hh\") pod \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") "
Sep 10 15:48:01.722575 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.722497    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-token\") pod \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\" (UID: \"26e02ee6-5f5a-4568-9508-4ef41b7163d5\") "
Sep 10 15:48:01.723665 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.723254    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "26e02ee6-5f5a-4568-9508-4ef41b7163d5" (UID: "26e02ee6-5f5a-4568-9508-4ef41b7163d5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:48:01.723665 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.723507    2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-csr-ca-configmap\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:48:01.725543 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.725506    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "26e02ee6-5f5a-4568-9508-4ef41b7163d5" (UID: "26e02ee6-5f5a-4568-9508-4ef41b7163d5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:48:01.726723 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.726687    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e02ee6-5f5a-4568-9508-4ef41b7163d5-cacerts" (OuterVolumeSpecName: "cacerts") pod "26e02ee6-5f5a-4568-9508-4ef41b7163d5" (UID: "26e02ee6-5f5a-4568-9508-4ef41b7163d5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:48:01.727249 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.727210    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "26e02ee6-5f5a-4568-9508-4ef41b7163d5" (UID: "26e02ee6-5f5a-4568-9508-4ef41b7163d5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:48:01.727439 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.727401    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e02ee6-5f5a-4568-9508-4ef41b7163d5-local-certs" (OuterVolumeSpecName: "local-certs") pod "26e02ee6-5f5a-4568-9508-4ef41b7163d5" (UID: "26e02ee6-5f5a-4568-9508-4ef41b7163d5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:48:01.729128 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.729104    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-token" (OuterVolumeSpecName: "istio-token") pod "26e02ee6-5f5a-4568-9508-4ef41b7163d5" (UID: "26e02ee6-5f5a-4568-9508-4ef41b7163d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:48:01.729128 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.729104    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e02ee6-5f5a-4568-9508-4ef41b7163d5-kube-api-access-cf6hh" (OuterVolumeSpecName: "kube-api-access-cf6hh") pod "26e02ee6-5f5a-4568-9508-4ef41b7163d5" (UID: "26e02ee6-5f5a-4568-9508-4ef41b7163d5"). InnerVolumeSpecName "kube-api-access-cf6hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:48:01.824406 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.824365    2564 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-kubeconfig\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:48:01.824406 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.824396    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf6hh\" (UniqueName: \"kubernetes.io/projected/26e02ee6-5f5a-4568-9508-4ef41b7163d5-kube-api-access-cf6hh\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:48:01.824615 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.824420    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-token\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:48:01.824615 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.824433    2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26e02ee6-5f5a-4568-9508-4ef41b7163d5-istio-csr-dns-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:48:01.824615 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.824446    2564 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e02ee6-5f5a-4568-9508-4ef41b7163d5-local-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:48:01.824615 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:01.824457    2564 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26e02ee6-5f5a-4568-9508-4ef41b7163d5-cacerts\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:48:02.233214 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:02.233149    2564 generic.go:356] "Generic (PLEG): container finished" podID="26e02ee6-5f5a-4568-9508-4ef41b7163d5" containerID="d6baafdf7c2f91b18b05ebc5a1deec46126df5343c1cbe4a1d0d631275df4fcd" exitCode=0
Sep 10 15:48:02.233478 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:02.233232    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"
Sep 10 15:48:02.233478 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:02.233272    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7" event={"ID":"26e02ee6-5f5a-4568-9508-4ef41b7163d5","Type":"ContainerDied","Data":"d6baafdf7c2f91b18b05ebc5a1deec46126df5343c1cbe4a1d0d631275df4fcd"}
Sep 10 15:48:02.233478 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:02.233334    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7" event={"ID":"26e02ee6-5f5a-4568-9508-4ef41b7163d5","Type":"ContainerDied","Data":"173c8bef46a0c85a9cd9bdf50bf773afdacafe18afbb3b8e5935c9b1160a7be7"}
Sep 10 15:48:02.233478 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:02.233361    2564 scope.go:117] "RemoveContainer" containerID="d6baafdf7c2f91b18b05ebc5a1deec46126df5343c1cbe4a1d0d631275df4fcd"
Sep 10 15:48:02.259981 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:02.259954    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"]
Sep 10 15:48:02.264579 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:02.264550    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xmvc7"]
Sep 10 15:48:03.349068 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:03.348981    2564 scope.go:117] "RemoveContainer" containerID="d6baafdf7c2f91b18b05ebc5a1deec46126df5343c1cbe4a1d0d631275df4fcd"
Sep 10 15:48:03.350061 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:48:03.350022    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6baafdf7c2f91b18b05ebc5a1deec46126df5343c1cbe4a1d0d631275df4fcd\": container with ID starting with d6baafdf7c2f91b18b05ebc5a1deec46126df5343c1cbe4a1d0d631275df4fcd not found: ID does not exist" containerID="d6baafdf7c2f91b18b05ebc5a1deec46126df5343c1cbe4a1d0d631275df4fcd"
Sep 10 15:48:03.350207 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:03.350071    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6baafdf7c2f91b18b05ebc5a1deec46126df5343c1cbe4a1d0d631275df4fcd"} err="failed to get container status \"d6baafdf7c2f91b18b05ebc5a1deec46126df5343c1cbe4a1d0d631275df4fcd\": rpc error: code = NotFound desc = could not find container \"d6baafdf7c2f91b18b05ebc5a1deec46126df5343c1cbe4a1d0d631275df4fcd\": container with ID starting with d6baafdf7c2f91b18b05ebc5a1deec46126df5343c1cbe4a1d0d631275df4fcd not found: ID does not exist"
Sep 10 15:48:03.671761 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:03.671728    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26e02ee6-5f5a-4568-9508-4ef41b7163d5" path="/var/lib/kubelet/pods/26e02ee6-5f5a-4568-9508-4ef41b7163d5/volumes"
Sep 10 15:48:04.245756 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:04.245724    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mb7mt" event={"ID":"69c36949-a1a5-4c1b-a250-4a08cb639da3","Type":"ContainerStarted","Data":"0f71ad352d82a78a78e7c1b2a1210e5bd20faa7b95064c66f652433e582a7d10"}
Sep 10 15:48:04.245972 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:04.245923    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mb7mt"
Sep 10 15:48:04.247377 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:04.247356    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vsdlt" event={"ID":"ab9b3bee-e44b-4a95-a430-7a712f5c323b","Type":"ContainerStarted","Data":"d87060c6ee7fbde4172c587b57f4cd2f07119e3e4d0337f815514c7f9ea52c0f"}
Sep 10 15:48:04.247500 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:04.247489    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vsdlt"
Sep 10 15:48:04.264315 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:04.264268    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mb7mt" podStartSLOduration=2.334170009 podStartE2EDuration="5.264255445s" totalImagesPullingTime="2.930085436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:48:04.262602316 +0000 UTC m=+363.066921577" watchObservedRunningTime="2026-09-10 15:48:04.264255445 +0000 UTC m=+363.068574706"
Sep 10 15:48:04.280177 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:04.280118    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vsdlt" podStartSLOduration=1.7776861940000002 podStartE2EDuration="5.280102827s" totalImagesPullingTime="3.502416633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:48:04.279151254 +0000 UTC m=+363.083470528" watchObservedRunningTime="2026-09-10 15:48:04.280102827 +0000 UTC m=+363.084422087"
Sep 10 15:48:05.193975 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:05.193947    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-2rkvm"
Sep 10 15:48:35.255252 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:35.255217    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vsdlt"
Sep 10 15:48:35.258087 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:48:35.258065    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mb7mt"
Sep 10 15:49:10.937840 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:10.937807    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"]
Sep 10 15:49:10.938284 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:10.938139    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e02ee6-5f5a-4568-9508-4ef41b7163d5" containerName="discovery"
Sep 10 15:49:10.938284 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:10.938150    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e02ee6-5f5a-4568-9508-4ef41b7163d5" containerName="discovery"
Sep 10 15:49:10.938284 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:10.938236    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e02ee6-5f5a-4568-9508-4ef41b7163d5" containerName="discovery"
Sep 10 15:49:10.940578 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:10.940562    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:10.944491 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:10.944468    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 15:49:10.944643 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:10.944568    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-w9tpd\""
Sep 10 15:49:10.945306 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:10.945292    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 15:49:10.972864 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:10.972810    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"]
Sep 10 15:49:11.043022 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:11.042988    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wffhz\" (UniqueName: \"kubernetes.io/projected/04ac4d67-cace-4288-a963-90b21295e9d2-kube-api-access-wffhz\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d7wnz\" (UID: \"04ac4d67-cace-4288-a963-90b21295e9d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:11.043229 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:11.043046    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/04ac4d67-cace-4288-a963-90b21295e9d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d7wnz\" (UID: \"04ac4d67-cace-4288-a963-90b21295e9d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:11.144281 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:11.144235    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wffhz\" (UniqueName: \"kubernetes.io/projected/04ac4d67-cace-4288-a963-90b21295e9d2-kube-api-access-wffhz\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d7wnz\" (UID: \"04ac4d67-cace-4288-a963-90b21295e9d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:11.144457 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:11.144300    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/04ac4d67-cace-4288-a963-90b21295e9d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d7wnz\" (UID: \"04ac4d67-cace-4288-a963-90b21295e9d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:11.144697 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:11.144681    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/04ac4d67-cace-4288-a963-90b21295e9d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d7wnz\" (UID: \"04ac4d67-cace-4288-a963-90b21295e9d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:11.157358 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:11.157333    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wffhz\" (UniqueName: \"kubernetes.io/projected/04ac4d67-cace-4288-a963-90b21295e9d2-kube-api-access-wffhz\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d7wnz\" (UID: \"04ac4d67-cace-4288-a963-90b21295e9d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:11.251922 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:11.251827    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:11.535901 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:11.535872    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"]
Sep 10 15:49:11.539448 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:49:11.539417    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ac4d67_cace_4288_a963_90b21295e9d2.slice/crio-3ed33093aae9f9b9c19e0f251ff91586afe2c0ade07b3361d6fa5d9f0ebd7166 WatchSource:0}: Error finding container 3ed33093aae9f9b9c19e0f251ff91586afe2c0ade07b3361d6fa5d9f0ebd7166: Status 404 returned error can't find the container with id 3ed33093aae9f9b9c19e0f251ff91586afe2c0ade07b3361d6fa5d9f0ebd7166
Sep 10 15:49:11.558634 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:11.558603    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz" event={"ID":"04ac4d67-cace-4288-a963-90b21295e9d2","Type":"ContainerStarted","Data":"3ed33093aae9f9b9c19e0f251ff91586afe2c0ade07b3361d6fa5d9f0ebd7166"}
Sep 10 15:49:12.930392 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:12.930195    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-76pts"]
Sep 10 15:49:12.935029 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:12.935001    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-76pts"
Sep 10 15:49:12.937417 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:12.937390    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-824vk\""
Sep 10 15:49:12.937961 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:12.937694    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 10 15:49:12.945887 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:12.945606    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-76pts"]
Sep 10 15:49:12.962144 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:12.961683    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvvtf\" (UniqueName: \"kubernetes.io/projected/8479fbe4-e17d-431b-9140-1da1df8e2f64-kube-api-access-qvvtf\") pod \"dns-operator-controller-manager-7b84cd7c9c-76pts\" (UID: \"8479fbe4-e17d-431b-9140-1da1df8e2f64\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-76pts"
Sep 10 15:49:13.062672 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:13.062626    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvvtf\" (UniqueName: \"kubernetes.io/projected/8479fbe4-e17d-431b-9140-1da1df8e2f64-kube-api-access-qvvtf\") pod \"dns-operator-controller-manager-7b84cd7c9c-76pts\" (UID: \"8479fbe4-e17d-431b-9140-1da1df8e2f64\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-76pts"
Sep 10 15:49:13.077853 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:13.077784    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvvtf\" (UniqueName: \"kubernetes.io/projected/8479fbe4-e17d-431b-9140-1da1df8e2f64-kube-api-access-qvvtf\") pod \"dns-operator-controller-manager-7b84cd7c9c-76pts\" (UID: \"8479fbe4-e17d-431b-9140-1da1df8e2f64\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-76pts"
Sep 10 15:49:13.251678 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:13.251593    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-76pts"
Sep 10 15:49:13.900755 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:13.900680    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-76pts"]
Sep 10 15:49:13.903601 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:49:13.903558    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8479fbe4_e17d_431b_9140_1da1df8e2f64.slice/crio-de41916c64532b584245a11e2e7b41f4dff8f9f0bdcdf05c93c2cd4cb9b51315 WatchSource:0}: Error finding container de41916c64532b584245a11e2e7b41f4dff8f9f0bdcdf05c93c2cd4cb9b51315: Status 404 returned error can't find the container with id de41916c64532b584245a11e2e7b41f4dff8f9f0bdcdf05c93c2cd4cb9b51315
Sep 10 15:49:14.580504 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:14.580217    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-76pts" event={"ID":"8479fbe4-e17d-431b-9140-1da1df8e2f64","Type":"ContainerStarted","Data":"de41916c64532b584245a11e2e7b41f4dff8f9f0bdcdf05c93c2cd4cb9b51315"}
Sep 10 15:49:15.559529 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:15.559494    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-csmdr"]
Sep 10 15:49:15.562268 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:15.562247    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-csmdr"
Sep 10 15:49:15.564447 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:15.564424    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-gj4hf\""
Sep 10 15:49:15.574345 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:15.574295    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-csmdr"]
Sep 10 15:49:15.589033 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:15.588994    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz" event={"ID":"04ac4d67-cace-4288-a963-90b21295e9d2","Type":"ContainerStarted","Data":"ab059a3cdb004edf885a09a10575f72ff5c032ad4f0c7f466062ab3de1e203cd"}
Sep 10 15:49:15.589636 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:15.589612    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:15.589831 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:15.589633    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6g5k\" (UniqueName: \"kubernetes.io/projected/149012af-8a86-480b-b7b0-49a4cacf0aea-kube-api-access-d6g5k\") pod \"authorino-operator-76b478bd78-csmdr\" (UID: \"149012af-8a86-480b-b7b0-49a4cacf0aea\") " pod="kuadrant-system/authorino-operator-76b478bd78-csmdr"
Sep 10 15:49:15.614658 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:15.614591    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz" podStartSLOduration=1.704789766 podStartE2EDuration="5.614568843s" totalImagesPullingTime="3.909779077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:49:15.612473086 +0000 UTC m=+434.416792349" watchObservedRunningTime="2026-09-10 15:49:15.614568843 +0000 UTC m=+434.418888107"
Sep 10 15:49:15.690916 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:15.690883    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6g5k\" (UniqueName: \"kubernetes.io/projected/149012af-8a86-480b-b7b0-49a4cacf0aea-kube-api-access-d6g5k\") pod \"authorino-operator-76b478bd78-csmdr\" (UID: \"149012af-8a86-480b-b7b0-49a4cacf0aea\") " pod="kuadrant-system/authorino-operator-76b478bd78-csmdr"
Sep 10 15:49:15.705628 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:15.705553    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6g5k\" (UniqueName: \"kubernetes.io/projected/149012af-8a86-480b-b7b0-49a4cacf0aea-kube-api-access-d6g5k\") pod \"authorino-operator-76b478bd78-csmdr\" (UID: \"149012af-8a86-480b-b7b0-49a4cacf0aea\") " pod="kuadrant-system/authorino-operator-76b478bd78-csmdr"
Sep 10 15:49:15.876741 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:15.876701    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-csmdr"
Sep 10 15:49:16.170995 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:16.170962    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-csmdr"]
Sep 10 15:49:16.345924 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:49:16.345891    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod149012af_8a86_480b_b7b0_49a4cacf0aea.slice/crio-91e8d972df7c841018553e3f11f6089288fb5370e4dd96f6f83c1ebfe392d2f5 WatchSource:0}: Error finding container 91e8d972df7c841018553e3f11f6089288fb5370e4dd96f6f83c1ebfe392d2f5: Status 404 returned error can't find the container with id 91e8d972df7c841018553e3f11f6089288fb5370e4dd96f6f83c1ebfe392d2f5
Sep 10 15:49:16.599200 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:16.599140    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-csmdr" event={"ID":"149012af-8a86-480b-b7b0-49a4cacf0aea","Type":"ContainerStarted","Data":"91e8d972df7c841018553e3f11f6089288fb5370e4dd96f6f83c1ebfe392d2f5"}
Sep 10 15:49:16.601280 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:16.601246    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-76pts" event={"ID":"8479fbe4-e17d-431b-9140-1da1df8e2f64","Type":"ContainerStarted","Data":"8a45c5b41f7e7a5bb8f5c2e0825114b838081d17a749f9e7a5bb225f423f3736"}
Sep 10 15:49:16.625927 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:16.625874    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-76pts" podStartSLOduration=2.1542134649999998 podStartE2EDuration="4.625856924s" totalImagesPullingTime="2.471643459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:49:16.624514615 +0000 UTC m=+435.428833876" watchObservedRunningTime="2026-09-10 15:49:16.625856924 +0000 UTC m=+435.430176185"
Sep 10 15:49:17.610186 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:17.608364    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-76pts"
Sep 10 15:49:17.980718 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:17.980684    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"]
Sep 10 15:49:17.983276 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:17.983253    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"
Sep 10 15:49:17.985720 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:17.985698    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-vw4bd\""
Sep 10 15:49:18.010912 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:18.010877    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"]
Sep 10 15:49:18.013343 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:18.013313    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hqs6\" (UniqueName: \"kubernetes.io/projected/b34df01d-a2de-4bd3-ba8a-b2c5028d0da8-kube-api-access-6hqs6\") pod \"limitador-operator-controller-manager-74495c69f8-jwmx6\" (UID: \"b34df01d-a2de-4bd3-ba8a-b2c5028d0da8\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"
Sep 10 15:49:18.113934 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:18.113890    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hqs6\" (UniqueName: \"kubernetes.io/projected/b34df01d-a2de-4bd3-ba8a-b2c5028d0da8-kube-api-access-6hqs6\") pod \"limitador-operator-controller-manager-74495c69f8-jwmx6\" (UID: \"b34df01d-a2de-4bd3-ba8a-b2c5028d0da8\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"
Sep 10 15:49:18.136004 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:18.135974    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hqs6\" (UniqueName: \"kubernetes.io/projected/b34df01d-a2de-4bd3-ba8a-b2c5028d0da8-kube-api-access-6hqs6\") pod \"limitador-operator-controller-manager-74495c69f8-jwmx6\" (UID: \"b34df01d-a2de-4bd3-ba8a-b2c5028d0da8\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"
Sep 10 15:49:18.295285 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:18.295199    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"
Sep 10 15:49:18.580691 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:18.580667    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"]
Sep 10 15:49:18.582824 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:49:18.582792    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34df01d_a2de_4bd3_ba8a_b2c5028d0da8.slice/crio-cb20e5409c23f3b3df377047f40d4c635d3305d9c63214199c2ca788e5c3b2b7 WatchSource:0}: Error finding container cb20e5409c23f3b3df377047f40d4c635d3305d9c63214199c2ca788e5c3b2b7: Status 404 returned error can't find the container with id cb20e5409c23f3b3df377047f40d4c635d3305d9c63214199c2ca788e5c3b2b7
Sep 10 15:49:18.614763 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:18.614722    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-csmdr" event={"ID":"149012af-8a86-480b-b7b0-49a4cacf0aea","Type":"ContainerStarted","Data":"71b1a096f22d34620eb8b649e789c985746a99ffe2ab57f72c8eac9afd9d948b"}
Sep 10 15:49:18.615128 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:18.614785    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-csmdr"
Sep 10 15:49:18.616124 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:18.616104    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6" event={"ID":"b34df01d-a2de-4bd3-ba8a-b2c5028d0da8","Type":"ContainerStarted","Data":"cb20e5409c23f3b3df377047f40d4c635d3305d9c63214199c2ca788e5c3b2b7"}
Sep 10 15:49:18.647288 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:18.647231    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-csmdr" podStartSLOduration=1.972988467 podStartE2EDuration="3.647213032s" totalImagesPullingTime="1.674224565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:49:18.645072906 +0000 UTC m=+437.449392181" watchObservedRunningTime="2026-09-10 15:49:18.647213032 +0000 UTC m=+437.451532292"
Sep 10 15:49:20.628328 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:20.628291    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6" event={"ID":"b34df01d-a2de-4bd3-ba8a-b2c5028d0da8","Type":"ContainerStarted","Data":"af91748902e92bc3b64868c8dc0aa336738bc229d9ad225670c50c3f45ce7d7b"}
Sep 10 15:49:20.628807 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:20.628416    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"
Sep 10 15:49:20.651051 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:20.650996    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6" podStartSLOduration=2.143797457 podStartE2EDuration="3.650979649s" totalImagesPullingTime="1.507182192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:49:20.649395582 +0000 UTC m=+439.453714850" watchObservedRunningTime="2026-09-10 15:49:20.650979649 +0000 UTC m=+439.455298910"
Sep 10 15:49:27.610272 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:27.610194    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:28.618835 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:28.618802    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-76pts"
Sep 10 15:49:29.624273 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:29.624238    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-csmdr"
Sep 10 15:49:29.929046 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:29.929015    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"]
Sep 10 15:49:29.929269 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:29.929246    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz" podUID="04ac4d67-cace-4288-a963-90b21295e9d2" containerName="manager" containerID="cri-o://ab059a3cdb004edf885a09a10575f72ff5c032ad4f0c7f466062ab3de1e203cd" gracePeriod=2
Sep 10 15:49:29.939619 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:29.939592    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"]
Sep 10 15:49:29.947875 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:29.947835    2564 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"04ac4d67-cace-4288-a963-90b21295e9d2\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-d7wnz\": pods \"kuadrant-operator-controller-manager-76f8f686bf-d7wnz\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:29.956666 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:29.956640    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"]
Sep 10 15:49:29.956907 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:29.956880    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6" podUID="b34df01d-a2de-4bd3-ba8a-b2c5028d0da8" containerName="manager" containerID="cri-o://af91748902e92bc3b64868c8dc0aa336738bc229d9ad225670c50c3f45ce7d7b" gracePeriod=2
Sep 10 15:49:29.958975 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:29.958944    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-d7wnz\" is forbidden: User \"system:node:ip-10-0-128-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-145.ec2.internal' and this object" podUID="04ac4d67-cace-4288-a963-90b21295e9d2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:29.959194 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:29.959066    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"
Sep 10 15:49:29.973315 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:29.973288    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w"]
Sep 10 15:49:29.973778 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:29.973758    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04ac4d67-cace-4288-a963-90b21295e9d2" containerName="manager"
Sep 10 15:49:29.973778 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:29.973780    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="04ac4d67-cace-4288-a963-90b21295e9d2" containerName="manager"
Sep 10 15:49:29.973916 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:29.973895    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="04ac4d67-cace-4288-a963-90b21295e9d2" containerName="manager"
Sep 10 15:49:29.976174 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:29.976140    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"]
Sep 10 15:49:29.976394 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:29.976344    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w"
Sep 10 15:49:29.980838 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:29.980809    2564 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"b34df01d-a2de-4bd3-ba8a-b2c5028d0da8\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-jwmx6\": pods \"limitador-operator-controller-manager-74495c69f8-jwmx6\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"
Sep 10 15:49:29.982717 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:29.982688    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-d7wnz\" is forbidden: User \"system:node:ip-10-0-128-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-145.ec2.internal' and this object" podUID="04ac4d67-cace-4288-a963-90b21295e9d2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:29.984589 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:29.984447    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-jwmx6\" is forbidden: User \"system:node:ip-10-0-128-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-145.ec2.internal' and this object" podUID="b34df01d-a2de-4bd3-ba8a-b2c5028d0da8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"
Sep 10 15:49:29.991083 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:29.991063    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2868c"]
Sep 10 15:49:29.991443 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:29.991429    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b34df01d-a2de-4bd3-ba8a-b2c5028d0da8" containerName="manager"
Sep 10 15:49:29.991519 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:29.991447    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b34df01d-a2de-4bd3-ba8a-b2c5028d0da8" containerName="manager"
Sep 10 15:49:29.991565 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:29.991532    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b34df01d-a2de-4bd3-ba8a-b2c5028d0da8" containerName="manager"
Sep 10 15:49:29.993435 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:29.993419    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2868c"
Sep 10 15:49:29.999329 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:29.999306    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w"]
Sep 10 15:49:30.020011 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.019987    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2868c"]
Sep 10 15:49:30.029311 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:30.029280    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-d7wnz\" is forbidden: User \"system:node:ip-10-0-128-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-145.ec2.internal' and this object" podUID="04ac4d67-cace-4288-a963-90b21295e9d2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:30.031205 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:30.031181    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-jwmx6\" is forbidden: User \"system:node:ip-10-0-128-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-145.ec2.internal' and this object" podUID="b34df01d-a2de-4bd3-ba8a-b2c5028d0da8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"
Sep 10 15:49:30.124261 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.124217    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/42329abf-711f-4962-a0a2-953836ecadc3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wzq9w\" (UID: \"42329abf-711f-4962-a0a2-953836ecadc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w"
Sep 10 15:49:30.124442 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.124307    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvxts\" (UniqueName: \"kubernetes.io/projected/42329abf-711f-4962-a0a2-953836ecadc3-kube-api-access-bvxts\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wzq9w\" (UID: \"42329abf-711f-4962-a0a2-953836ecadc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w"
Sep 10 15:49:30.124442 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.124334    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4jhw\" (UniqueName: \"kubernetes.io/projected/20c43a64-1f3d-4273-b3a9-3e2704dde2db-kube-api-access-c4jhw\") pod \"limitador-operator-controller-manager-74495c69f8-2868c\" (UID: \"20c43a64-1f3d-4273-b3a9-3e2704dde2db\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2868c"
Sep 10 15:49:30.175450 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.175428    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:30.177673 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:30.177648    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-d7wnz\" is forbidden: User \"system:node:ip-10-0-128-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-145.ec2.internal' and this object" podUID="04ac4d67-cace-4288-a963-90b21295e9d2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:30.178694 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.178681    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"
Sep 10 15:49:30.179475 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:30.179432    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-jwmx6\" is forbidden: User \"system:node:ip-10-0-128-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-145.ec2.internal' and this object" podUID="b34df01d-a2de-4bd3-ba8a-b2c5028d0da8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"
Sep 10 15:49:30.181379 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:30.181359    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-d7wnz\" is forbidden: User \"system:node:ip-10-0-128-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-145.ec2.internal' and this object" podUID="04ac4d67-cace-4288-a963-90b21295e9d2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:30.183199 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:30.183148    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-jwmx6\" is forbidden: User \"system:node:ip-10-0-128-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-145.ec2.internal' and this object" podUID="b34df01d-a2de-4bd3-ba8a-b2c5028d0da8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"
Sep 10 15:49:30.225026 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.224995    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvxts\" (UniqueName: \"kubernetes.io/projected/42329abf-711f-4962-a0a2-953836ecadc3-kube-api-access-bvxts\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wzq9w\" (UID: \"42329abf-711f-4962-a0a2-953836ecadc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w"
Sep 10 15:49:30.225179 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.225038    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4jhw\" (UniqueName: \"kubernetes.io/projected/20c43a64-1f3d-4273-b3a9-3e2704dde2db-kube-api-access-c4jhw\") pod \"limitador-operator-controller-manager-74495c69f8-2868c\" (UID: \"20c43a64-1f3d-4273-b3a9-3e2704dde2db\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2868c"
Sep 10 15:49:30.225311 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.225288    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/42329abf-711f-4962-a0a2-953836ecadc3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wzq9w\" (UID: \"42329abf-711f-4962-a0a2-953836ecadc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w"
Sep 10 15:49:30.225674 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.225656    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/42329abf-711f-4962-a0a2-953836ecadc3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wzq9w\" (UID: \"42329abf-711f-4962-a0a2-953836ecadc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w"
Sep 10 15:49:30.239685 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.239658    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvxts\" (UniqueName: \"kubernetes.io/projected/42329abf-711f-4962-a0a2-953836ecadc3-kube-api-access-bvxts\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wzq9w\" (UID: \"42329abf-711f-4962-a0a2-953836ecadc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w"
Sep 10 15:49:30.239822 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.239798    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4jhw\" (UniqueName: \"kubernetes.io/projected/20c43a64-1f3d-4273-b3a9-3e2704dde2db-kube-api-access-c4jhw\") pod \"limitador-operator-controller-manager-74495c69f8-2868c\" (UID: \"20c43a64-1f3d-4273-b3a9-3e2704dde2db\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2868c"
Sep 10 15:49:30.326176 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.326135    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/04ac4d67-cace-4288-a963-90b21295e9d2-extensions-socket-volume\") pod \"04ac4d67-cace-4288-a963-90b21295e9d2\" (UID: \"04ac4d67-cace-4288-a963-90b21295e9d2\") "
Sep 10 15:49:30.326362 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.326199    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hqs6\" (UniqueName: \"kubernetes.io/projected/b34df01d-a2de-4bd3-ba8a-b2c5028d0da8-kube-api-access-6hqs6\") pod \"b34df01d-a2de-4bd3-ba8a-b2c5028d0da8\" (UID: \"b34df01d-a2de-4bd3-ba8a-b2c5028d0da8\") "
Sep 10 15:49:30.326362 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.326232    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wffhz\" (UniqueName: \"kubernetes.io/projected/04ac4d67-cace-4288-a963-90b21295e9d2-kube-api-access-wffhz\") pod \"04ac4d67-cace-4288-a963-90b21295e9d2\" (UID: \"04ac4d67-cace-4288-a963-90b21295e9d2\") "
Sep 10 15:49:30.326609 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.326585    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ac4d67-cace-4288-a963-90b21295e9d2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "04ac4d67-cace-4288-a963-90b21295e9d2" (UID: "04ac4d67-cace-4288-a963-90b21295e9d2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:49:30.328590 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.328564    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ac4d67-cace-4288-a963-90b21295e9d2-kube-api-access-wffhz" (OuterVolumeSpecName: "kube-api-access-wffhz") pod "04ac4d67-cace-4288-a963-90b21295e9d2" (UID: "04ac4d67-cace-4288-a963-90b21295e9d2"). InnerVolumeSpecName "kube-api-access-wffhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:49:30.328685 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.328592    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34df01d-a2de-4bd3-ba8a-b2c5028d0da8-kube-api-access-6hqs6" (OuterVolumeSpecName: "kube-api-access-6hqs6") pod "b34df01d-a2de-4bd3-ba8a-b2c5028d0da8" (UID: "b34df01d-a2de-4bd3-ba8a-b2c5028d0da8"). InnerVolumeSpecName "kube-api-access-6hqs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:49:30.371439 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.371404    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w"
Sep 10 15:49:30.377792 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.377766    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2868c"
Sep 10 15:49:30.428063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.428033    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/04ac4d67-cace-4288-a963-90b21295e9d2-extensions-socket-volume\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:49:30.428185 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.428071    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hqs6\" (UniqueName: \"kubernetes.io/projected/b34df01d-a2de-4bd3-ba8a-b2c5028d0da8-kube-api-access-6hqs6\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:49:30.428185 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.428086    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wffhz\" (UniqueName: \"kubernetes.io/projected/04ac4d67-cace-4288-a963-90b21295e9d2-kube-api-access-wffhz\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:49:30.650998 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.650976    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w"]
Sep 10 15:49:30.653733 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:49:30.653570    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42329abf_711f_4962_a0a2_953836ecadc3.slice/crio-866695afb58a46176c8b36b73a37faab7c42d8c036a36b9d9066950627c88aa7 WatchSource:0}: Error finding container 866695afb58a46176c8b36b73a37faab7c42d8c036a36b9d9066950627c88aa7: Status 404 returned error can't find the container with id 866695afb58a46176c8b36b73a37faab7c42d8c036a36b9d9066950627c88aa7
Sep 10 15:49:30.653876 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.653855    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2868c"]
Sep 10 15:49:30.656279 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:49:30.656258    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c43a64_1f3d_4273_b3a9_3e2704dde2db.slice/crio-1909b11c9ebde692a51768137e000613ae71cb78fe0929ce6861f9371d50e8f4 WatchSource:0}: Error finding container 1909b11c9ebde692a51768137e000613ae71cb78fe0929ce6861f9371d50e8f4: Status 404 returned error can't find the container with id 1909b11c9ebde692a51768137e000613ae71cb78fe0929ce6861f9371d50e8f4
Sep 10 15:49:30.680011 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.679957    2564 generic.go:356] "Generic (PLEG): container finished" podID="b34df01d-a2de-4bd3-ba8a-b2c5028d0da8" containerID="af91748902e92bc3b64868c8dc0aa336738bc229d9ad225670c50c3f45ce7d7b" exitCode=0
Sep 10 15:49:30.680011 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.680000    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"
Sep 10 15:49:30.680137 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.680044    2564 scope.go:117] "RemoveContainer" containerID="af91748902e92bc3b64868c8dc0aa336738bc229d9ad225670c50c3f45ce7d7b"
Sep 10 15:49:30.681677 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.681655    2564 generic.go:356] "Generic (PLEG): container finished" podID="04ac4d67-cace-4288-a963-90b21295e9d2" containerID="ab059a3cdb004edf885a09a10575f72ff5c032ad4f0c7f466062ab3de1e203cd" exitCode=0
Sep 10 15:49:30.681763 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.681689    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:30.685340 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:30.682496    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-d7wnz\" is forbidden: User \"system:node:ip-10-0-128-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-145.ec2.internal' and this object" podUID="04ac4d67-cace-4288-a963-90b21295e9d2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:30.685340 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:30.684352    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-jwmx6\" is forbidden: User \"system:node:ip-10-0-128-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-145.ec2.internal' and this object" podUID="b34df01d-a2de-4bd3-ba8a-b2c5028d0da8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"
Sep 10 15:49:30.689151 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:30.686123    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-d7wnz\" is forbidden: User \"system:node:ip-10-0-128-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-145.ec2.internal' and this object" podUID="04ac4d67-cace-4288-a963-90b21295e9d2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:30.689151 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:30.687940    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-jwmx6\" is forbidden: User \"system:node:ip-10-0-128-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-145.ec2.internal' and this object" podUID="b34df01d-a2de-4bd3-ba8a-b2c5028d0da8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"
Sep 10 15:49:30.690198 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.690031    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2868c" event={"ID":"20c43a64-1f3d-4273-b3a9-3e2704dde2db","Type":"ContainerStarted","Data":"1909b11c9ebde692a51768137e000613ae71cb78fe0929ce6861f9371d50e8f4"}
Sep 10 15:49:30.692430 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.692388    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w" event={"ID":"42329abf-711f-4962-a0a2-953836ecadc3","Type":"ContainerStarted","Data":"866695afb58a46176c8b36b73a37faab7c42d8c036a36b9d9066950627c88aa7"}
Sep 10 15:49:30.696115 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.696095    2564 scope.go:117] "RemoveContainer" containerID="af91748902e92bc3b64868c8dc0aa336738bc229d9ad225670c50c3f45ce7d7b"
Sep 10 15:49:30.696426 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:30.696408    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af91748902e92bc3b64868c8dc0aa336738bc229d9ad225670c50c3f45ce7d7b\": container with ID starting with af91748902e92bc3b64868c8dc0aa336738bc229d9ad225670c50c3f45ce7d7b not found: ID does not exist" containerID="af91748902e92bc3b64868c8dc0aa336738bc229d9ad225670c50c3f45ce7d7b"
Sep 10 15:49:30.696467 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.696434    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af91748902e92bc3b64868c8dc0aa336738bc229d9ad225670c50c3f45ce7d7b"} err="failed to get container status \"af91748902e92bc3b64868c8dc0aa336738bc229d9ad225670c50c3f45ce7d7b\": rpc error: code = NotFound desc = could not find container \"af91748902e92bc3b64868c8dc0aa336738bc229d9ad225670c50c3f45ce7d7b\": container with ID starting with af91748902e92bc3b64868c8dc0aa336738bc229d9ad225670c50c3f45ce7d7b not found: ID does not exist"
Sep 10 15:49:30.696467 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.696449    2564 scope.go:117] "RemoveContainer" containerID="ab059a3cdb004edf885a09a10575f72ff5c032ad4f0c7f466062ab3de1e203cd"
Sep 10 15:49:30.706570 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.706548    2564 scope.go:117] "RemoveContainer" containerID="ab059a3cdb004edf885a09a10575f72ff5c032ad4f0c7f466062ab3de1e203cd"
Sep 10 15:49:30.706852 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:30.706833    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab059a3cdb004edf885a09a10575f72ff5c032ad4f0c7f466062ab3de1e203cd\": container with ID starting with ab059a3cdb004edf885a09a10575f72ff5c032ad4f0c7f466062ab3de1e203cd not found: ID does not exist" containerID="ab059a3cdb004edf885a09a10575f72ff5c032ad4f0c7f466062ab3de1e203cd"
Sep 10 15:49:30.706918 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:30.706857    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab059a3cdb004edf885a09a10575f72ff5c032ad4f0c7f466062ab3de1e203cd"} err="failed to get container status \"ab059a3cdb004edf885a09a10575f72ff5c032ad4f0c7f466062ab3de1e203cd\": rpc error: code = NotFound desc = could not find container \"ab059a3cdb004edf885a09a10575f72ff5c032ad4f0c7f466062ab3de1e203cd\": container with ID starting with ab059a3cdb004edf885a09a10575f72ff5c032ad4f0c7f466062ab3de1e203cd not found: ID does not exist"
Sep 10 15:49:30.710647 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:30.710621    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-d7wnz\" is forbidden: User \"system:node:ip-10-0-128-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-145.ec2.internal' and this object" podUID="04ac4d67-cace-4288-a963-90b21295e9d2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:30.712695 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:30.712665    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-jwmx6\" is forbidden: User \"system:node:ip-10-0-128-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-145.ec2.internal' and this object" podUID="b34df01d-a2de-4bd3-ba8a-b2c5028d0da8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"
Sep 10 15:49:30.714622 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:30.714602    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-d7wnz\" is forbidden: User \"system:node:ip-10-0-128-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-145.ec2.internal' and this object" podUID="04ac4d67-cace-4288-a963-90b21295e9d2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:30.716216 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:30.716195    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-jwmx6\" is forbidden: User \"system:node:ip-10-0-128-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-145.ec2.internal' and this object" podUID="b34df01d-a2de-4bd3-ba8a-b2c5028d0da8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"
Sep 10 15:49:31.236447 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:31.236362    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx"]
Sep 10 15:49:31.239137 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:31.239114    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx"
Sep 10 15:49:31.241851 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:31.241823    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-d7wnz\" is forbidden: User \"system:node:ip-10-0-128-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-145.ec2.internal' and this object" podUID="04ac4d67-cace-4288-a963-90b21295e9d2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:31.243893 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:31.243864    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-jwmx6\" is forbidden: User \"system:node:ip-10-0-128-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-145.ec2.internal' and this object" podUID="b34df01d-a2de-4bd3-ba8a-b2c5028d0da8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"
Sep 10 15:49:31.253631 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:31.253606    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx"]
Sep 10 15:49:31.337298 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:31.337256    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6l8x\" (UniqueName: \"kubernetes.io/projected/a90ce5c4-9824-4886-9f8c-caacf5555176-kube-api-access-b6l8x\") pod \"kuadrant-operator-controller-manager-5769f99d9f-47bpx\" (UID: \"a90ce5c4-9824-4886-9f8c-caacf5555176\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx"
Sep 10 15:49:31.337467 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:31.337364    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a90ce5c4-9824-4886-9f8c-caacf5555176-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-47bpx\" (UID: \"a90ce5c4-9824-4886-9f8c-caacf5555176\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx"
Sep 10 15:49:31.438212 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:31.438152    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a90ce5c4-9824-4886-9f8c-caacf5555176-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-47bpx\" (UID: \"a90ce5c4-9824-4886-9f8c-caacf5555176\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx"
Sep 10 15:49:31.438369 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:31.438257    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6l8x\" (UniqueName: \"kubernetes.io/projected/a90ce5c4-9824-4886-9f8c-caacf5555176-kube-api-access-b6l8x\") pod \"kuadrant-operator-controller-manager-5769f99d9f-47bpx\" (UID: \"a90ce5c4-9824-4886-9f8c-caacf5555176\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx"
Sep 10 15:49:31.438580 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:31.438561    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a90ce5c4-9824-4886-9f8c-caacf5555176-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-47bpx\" (UID: \"a90ce5c4-9824-4886-9f8c-caacf5555176\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx"
Sep 10 15:49:31.450981 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:31.450957    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6l8x\" (UniqueName: \"kubernetes.io/projected/a90ce5c4-9824-4886-9f8c-caacf5555176-kube-api-access-b6l8x\") pod \"kuadrant-operator-controller-manager-5769f99d9f-47bpx\" (UID: \"a90ce5c4-9824-4886-9f8c-caacf5555176\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx"
Sep 10 15:49:31.550263 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:31.550178    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx"
Sep 10 15:49:31.670987 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:31.670955    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-d7wnz\" is forbidden: User \"system:node:ip-10-0-128-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-145.ec2.internal' and this object" podUID="04ac4d67-cace-4288-a963-90b21295e9d2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d7wnz"
Sep 10 15:49:31.672409 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:31.672389    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04ac4d67-cace-4288-a963-90b21295e9d2" path="/var/lib/kubelet/pods/04ac4d67-cace-4288-a963-90b21295e9d2/volumes"
Sep 10 15:49:31.672698 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:31.672687    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b34df01d-a2de-4bd3-ba8a-b2c5028d0da8" path="/var/lib/kubelet/pods/b34df01d-a2de-4bd3-ba8a-b2c5028d0da8/volumes"
Sep 10 15:49:31.673403 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:31.673384    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-jwmx6\" is forbidden: User \"system:node:ip-10-0-128-145.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-145.ec2.internal' and this object" podUID="b34df01d-a2de-4bd3-ba8a-b2c5028d0da8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jwmx6"
Sep 10 15:49:31.699970 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:31.699934    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w" event={"ID":"42329abf-711f-4962-a0a2-953836ecadc3","Type":"ContainerStarted","Data":"889a0d6035a2785d2c3199f020e7861813eff25bd0f68cd84742743f4331841c"}
Sep 10 15:49:31.700186 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:31.700012    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w"
Sep 10 15:49:31.704720 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:31.704689    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2868c" event={"ID":"20c43a64-1f3d-4273-b3a9-3e2704dde2db","Type":"ContainerStarted","Data":"420179416c7bc20f10c570118b34151d4d00b7455b1d3c978862f32bd0876a04"}
Sep 10 15:49:31.704873 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:31.704831    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2868c"
Sep 10 15:49:31.728890 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:31.728840    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w" podStartSLOduration=2.7288234129999998 podStartE2EDuration="2.728823413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:49:31.727131086 +0000 UTC m=+450.531450347" watchObservedRunningTime="2026-09-10 15:49:31.728823413 +0000 UTC m=+450.533142673"
Sep 10 15:49:31.761532 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:31.761477    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2868c" podStartSLOduration=2.761461712 podStartE2EDuration="2.761461712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:49:31.760357753 +0000 UTC m=+450.564677016" watchObservedRunningTime="2026-09-10 15:49:31.761461712 +0000 UTC m=+450.565780999"
Sep 10 15:49:31.846856 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:31.846781    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx"]
Sep 10 15:49:31.850635 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:49:31.850602    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90ce5c4_9824_4886_9f8c_caacf5555176.slice/crio-d768a78d9148ba1e362727f7e60879a8e48de502216b67de51190546a1c1b58f WatchSource:0}: Error finding container d768a78d9148ba1e362727f7e60879a8e48de502216b67de51190546a1c1b58f: Status 404 returned error can't find the container with id d768a78d9148ba1e362727f7e60879a8e48de502216b67de51190546a1c1b58f
Sep 10 15:49:32.711929 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:32.711887    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx" event={"ID":"a90ce5c4-9824-4886-9f8c-caacf5555176","Type":"ContainerStarted","Data":"9f8193d994ab74279e3f895e9c80c3d7d9c059329807bb07155ae814dc7c1607"}
Sep 10 15:49:32.712428 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:32.711956    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx"
Sep 10 15:49:32.712428 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:32.711982    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx" event={"ID":"a90ce5c4-9824-4886-9f8c-caacf5555176","Type":"ContainerStarted","Data":"d768a78d9148ba1e362727f7e60879a8e48de502216b67de51190546a1c1b58f"}
Sep 10 15:49:32.738985 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:32.738937    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx" podStartSLOduration=1.738919665 podStartE2EDuration="1.738919665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:49:32.737237362 +0000 UTC m=+451.541556623" watchObservedRunningTime="2026-09-10 15:49:32.738919665 +0000 UTC m=+451.543238932"
Sep 10 15:49:42.714370 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:42.714292    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w"
Sep 10 15:49:42.715061 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:42.714386    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2868c"
Sep 10 15:49:43.718972 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:43.718943    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx"
Sep 10 15:49:43.777352 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:43.777314    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w"]
Sep 10 15:49:43.777583 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:43.777531    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w" podUID="42329abf-711f-4962-a0a2-953836ecadc3" containerName="manager" containerID="cri-o://889a0d6035a2785d2c3199f020e7861813eff25bd0f68cd84742743f4331841c" gracePeriod=10
Sep 10 15:49:44.023665 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:44.023637    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w"
Sep 10 15:49:44.151069 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:44.151032    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvxts\" (UniqueName: \"kubernetes.io/projected/42329abf-711f-4962-a0a2-953836ecadc3-kube-api-access-bvxts\") pod \"42329abf-711f-4962-a0a2-953836ecadc3\" (UID: \"42329abf-711f-4962-a0a2-953836ecadc3\") "
Sep 10 15:49:44.151253 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:44.151108    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/42329abf-711f-4962-a0a2-953836ecadc3-extensions-socket-volume\") pod \"42329abf-711f-4962-a0a2-953836ecadc3\" (UID: \"42329abf-711f-4962-a0a2-953836ecadc3\") "
Sep 10 15:49:44.151485 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:44.151463    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42329abf-711f-4962-a0a2-953836ecadc3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "42329abf-711f-4962-a0a2-953836ecadc3" (UID: "42329abf-711f-4962-a0a2-953836ecadc3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:49:44.153297 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:44.153274    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42329abf-711f-4962-a0a2-953836ecadc3-kube-api-access-bvxts" (OuterVolumeSpecName: "kube-api-access-bvxts") pod "42329abf-711f-4962-a0a2-953836ecadc3" (UID: "42329abf-711f-4962-a0a2-953836ecadc3"). InnerVolumeSpecName "kube-api-access-bvxts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:49:44.252140 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:44.252067    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvxts\" (UniqueName: \"kubernetes.io/projected/42329abf-711f-4962-a0a2-953836ecadc3-kube-api-access-bvxts\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:49:44.252140 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:44.252093    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/42329abf-711f-4962-a0a2-953836ecadc3-extensions-socket-volume\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:49:44.775101 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:44.775069    2564 generic.go:356] "Generic (PLEG): container finished" podID="42329abf-711f-4962-a0a2-953836ecadc3" containerID="889a0d6035a2785d2c3199f020e7861813eff25bd0f68cd84742743f4331841c" exitCode=0
Sep 10 15:49:44.775521 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:44.775126    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w"
Sep 10 15:49:44.775521 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:44.775139    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w" event={"ID":"42329abf-711f-4962-a0a2-953836ecadc3","Type":"ContainerDied","Data":"889a0d6035a2785d2c3199f020e7861813eff25bd0f68cd84742743f4331841c"}
Sep 10 15:49:44.775521 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:44.775218    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w" event={"ID":"42329abf-711f-4962-a0a2-953836ecadc3","Type":"ContainerDied","Data":"866695afb58a46176c8b36b73a37faab7c42d8c036a36b9d9066950627c88aa7"}
Sep 10 15:49:44.775521 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:44.775241    2564 scope.go:117] "RemoveContainer" containerID="889a0d6035a2785d2c3199f020e7861813eff25bd0f68cd84742743f4331841c"
Sep 10 15:49:44.784053 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:44.783888    2564 scope.go:117] "RemoveContainer" containerID="889a0d6035a2785d2c3199f020e7861813eff25bd0f68cd84742743f4331841c"
Sep 10 15:49:44.784204 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:49:44.784180    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"889a0d6035a2785d2c3199f020e7861813eff25bd0f68cd84742743f4331841c\": container with ID starting with 889a0d6035a2785d2c3199f020e7861813eff25bd0f68cd84742743f4331841c not found: ID does not exist" containerID="889a0d6035a2785d2c3199f020e7861813eff25bd0f68cd84742743f4331841c"
Sep 10 15:49:44.784302 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:44.784209    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"889a0d6035a2785d2c3199f020e7861813eff25bd0f68cd84742743f4331841c"} err="failed to get container status \"889a0d6035a2785d2c3199f020e7861813eff25bd0f68cd84742743f4331841c\": rpc error: code = NotFound desc = could not find container \"889a0d6035a2785d2c3199f020e7861813eff25bd0f68cd84742743f4331841c\": container with ID starting with 889a0d6035a2785d2c3199f020e7861813eff25bd0f68cd84742743f4331841c not found: ID does not exist"
Sep 10 15:49:44.800666 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:44.800637    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w"]
Sep 10 15:49:44.804764 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:44.804741    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wzq9w"]
Sep 10 15:49:45.672376 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:49:45.672343    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42329abf-711f-4962-a0a2-953836ecadc3" path="/var/lib/kubelet/pods/42329abf-711f-4962-a0a2-953836ecadc3/volumes"
Sep 10 15:50:00.102776 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.102746    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"]
Sep 10 15:50:00.103282 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.103105    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42329abf-711f-4962-a0a2-953836ecadc3" containerName="manager"
Sep 10 15:50:00.103282 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.103117    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="42329abf-711f-4962-a0a2-953836ecadc3" containerName="manager"
Sep 10 15:50:00.103282 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.103217    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="42329abf-711f-4962-a0a2-953836ecadc3" containerName="manager"
Sep 10 15:50:00.105717 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.105697    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.108399 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.108375    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 15:50:00.108515 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.108431    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rnnk2\""
Sep 10 15:50:00.121370 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.121301    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"]
Sep 10 15:50:00.191852 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.191820    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knzd7\" (UniqueName: \"kubernetes.io/projected/5d6bce58-c1f0-46be-8110-c84cab62ff75-kube-api-access-knzd7\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.192020 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.191868    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6bce58-c1f0-46be-8110-c84cab62ff75-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.192020 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.191892    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6bce58-c1f0-46be-8110-c84cab62ff75-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.192020 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.191915    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6bce58-c1f0-46be-8110-c84cab62ff75-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.192020 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.191940    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6bce58-c1f0-46be-8110-c84cab62ff75-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.192020 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.191956    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6bce58-c1f0-46be-8110-c84cab62ff75-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.192020 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.191971    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6bce58-c1f0-46be-8110-c84cab62ff75-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.192253 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.192032    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6bce58-c1f0-46be-8110-c84cab62ff75-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.192253 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.192067    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6bce58-c1f0-46be-8110-c84cab62ff75-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.293011 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.292976    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knzd7\" (UniqueName: \"kubernetes.io/projected/5d6bce58-c1f0-46be-8110-c84cab62ff75-kube-api-access-knzd7\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.293213 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.293022    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6bce58-c1f0-46be-8110-c84cab62ff75-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.293213 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.293043    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6bce58-c1f0-46be-8110-c84cab62ff75-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.293213 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.293063    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6bce58-c1f0-46be-8110-c84cab62ff75-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.293213 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.293096    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6bce58-c1f0-46be-8110-c84cab62ff75-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.293213 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.293114    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6bce58-c1f0-46be-8110-c84cab62ff75-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.293213 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.293129    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6bce58-c1f0-46be-8110-c84cab62ff75-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.293213 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.293149    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6bce58-c1f0-46be-8110-c84cab62ff75-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.293213 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.293206    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6bce58-c1f0-46be-8110-c84cab62ff75-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.293624 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.293505    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6bce58-c1f0-46be-8110-c84cab62ff75-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.293682 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.293652    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6bce58-c1f0-46be-8110-c84cab62ff75-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.293682 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.293667    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6bce58-c1f0-46be-8110-c84cab62ff75-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.293789 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.293740    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6bce58-c1f0-46be-8110-c84cab62ff75-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.294012 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.293995    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6bce58-c1f0-46be-8110-c84cab62ff75-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.295507 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.295483    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6bce58-c1f0-46be-8110-c84cab62ff75-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.295814 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.295796    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6bce58-c1f0-46be-8110-c84cab62ff75-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.303889 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.303866    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6bce58-c1f0-46be-8110-c84cab62ff75-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.304521 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.304503    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knzd7\" (UniqueName: \"kubernetes.io/projected/5d6bce58-c1f0-46be-8110-c84cab62ff75-kube-api-access-knzd7\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wgwhz\" (UID: \"5d6bce58-c1f0-46be-8110-c84cab62ff75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.416005 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.415917    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:00.691251 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.691225    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"]
Sep 10 15:50:00.693362 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:50:00.693338    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6bce58_c1f0_46be_8110_c84cab62ff75.slice/crio-bcdce443b635237017d446ba6eca739079a137cf71b7f64743b9fad80346da32 WatchSource:0}: Error finding container bcdce443b635237017d446ba6eca739079a137cf71b7f64743b9fad80346da32: Status 404 returned error can't find the container with id bcdce443b635237017d446ba6eca739079a137cf71b7f64743b9fad80346da32
Sep 10 15:50:00.854358 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:00.854323    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz" event={"ID":"5d6bce58-c1f0-46be-8110-c84cab62ff75","Type":"ContainerStarted","Data":"bcdce443b635237017d446ba6eca739079a137cf71b7f64743b9fad80346da32"}
Sep 10 15:50:04.321666 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:04.321634    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:50:04.321913 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:04.321704    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:50:04.321913 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:04.321733    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:50:04.877816 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:04.877782    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz" event={"ID":"5d6bce58-c1f0-46be-8110-c84cab62ff75","Type":"ContainerStarted","Data":"daba2ded886f423927ef0759b990728fb15b16999145a973ffa934a0eca01907"}
Sep 10 15:50:04.902224 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:04.902143    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz" podStartSLOduration=1.276155663 podStartE2EDuration="4.902126942s" totalImagesPullingTime="3.625971279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:04.90113634 +0000 UTC m=+483.705455602" watchObservedRunningTime="2026-09-10 15:50:04.902126942 +0000 UTC m=+483.706446261"
Sep 10 15:50:05.416665 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:05.416631    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:05.421467 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:05.421442    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:05.883124 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:05.883093    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:05.884118 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:05.884096    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:50:05.949457 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:05.949429    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6j9zt"]
Sep 10 15:50:05.952080 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:05.952060    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-6j9zt"
Sep 10 15:50:05.954912 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:05.954894    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bphzg\""
Sep 10 15:50:05.954912 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:05.954902    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 15:50:05.967143 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:05.967118    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6j9zt"]
Sep 10 15:50:06.046202 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:06.046144    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9vc9\" (UniqueName: \"kubernetes.io/projected/06d00021-5706-4eb4-bdd8-50d30f5f1db1-kube-api-access-m9vc9\") pod \"limitador-limitador-8588749977-6j9zt\" (UID: \"06d00021-5706-4eb4-bdd8-50d30f5f1db1\") " pod="kuadrant-system/limitador-limitador-8588749977-6j9zt"
Sep 10 15:50:06.046363 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:06.046295    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/06d00021-5706-4eb4-bdd8-50d30f5f1db1-config-file\") pod \"limitador-limitador-8588749977-6j9zt\" (UID: \"06d00021-5706-4eb4-bdd8-50d30f5f1db1\") " pod="kuadrant-system/limitador-limitador-8588749977-6j9zt"
Sep 10 15:50:06.060958 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:06.060929    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6j9zt"]
Sep 10 15:50:06.147046 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:06.146944    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9vc9\" (UniqueName: \"kubernetes.io/projected/06d00021-5706-4eb4-bdd8-50d30f5f1db1-kube-api-access-m9vc9\") pod \"limitador-limitador-8588749977-6j9zt\" (UID: \"06d00021-5706-4eb4-bdd8-50d30f5f1db1\") " pod="kuadrant-system/limitador-limitador-8588749977-6j9zt"
Sep 10 15:50:06.147281 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:06.147061    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/06d00021-5706-4eb4-bdd8-50d30f5f1db1-config-file\") pod \"limitador-limitador-8588749977-6j9zt\" (UID: \"06d00021-5706-4eb4-bdd8-50d30f5f1db1\") " pod="kuadrant-system/limitador-limitador-8588749977-6j9zt"
Sep 10 15:50:06.147908 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:06.147881    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/06d00021-5706-4eb4-bdd8-50d30f5f1db1-config-file\") pod \"limitador-limitador-8588749977-6j9zt\" (UID: \"06d00021-5706-4eb4-bdd8-50d30f5f1db1\") " pod="kuadrant-system/limitador-limitador-8588749977-6j9zt"
Sep 10 15:50:06.159035 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:06.159011    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9vc9\" (UniqueName: \"kubernetes.io/projected/06d00021-5706-4eb4-bdd8-50d30f5f1db1-kube-api-access-m9vc9\") pod \"limitador-limitador-8588749977-6j9zt\" (UID: \"06d00021-5706-4eb4-bdd8-50d30f5f1db1\") " pod="kuadrant-system/limitador-limitador-8588749977-6j9zt"
Sep 10 15:50:06.262335 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:06.262300    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-6j9zt"
Sep 10 15:50:06.533183 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:06.533129    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6j9zt"]
Sep 10 15:50:06.535006 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:50:06.534975    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d00021_5706_4eb4_bdd8_50d30f5f1db1.slice/crio-e3d72c17cbae36ea4b305d08e4c59d302e019630b144c5e4ada460bd6a79d6c2 WatchSource:0}: Error finding container e3d72c17cbae36ea4b305d08e4c59d302e019630b144c5e4ada460bd6a79d6c2: Status 404 returned error can't find the container with id e3d72c17cbae36ea4b305d08e4c59d302e019630b144c5e4ada460bd6a79d6c2
Sep 10 15:50:06.888950 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:06.888865    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-6j9zt" event={"ID":"06d00021-5706-4eb4-bdd8-50d30f5f1db1","Type":"ContainerStarted","Data":"e3d72c17cbae36ea4b305d08e4c59d302e019630b144c5e4ada460bd6a79d6c2"}
Sep 10 15:50:06.891994 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:06.891970    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-n2dql"]
Sep 10 15:50:06.895608 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:06.895591    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-n2dql"
Sep 10 15:50:06.897690 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:06.897673    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fswpm\""
Sep 10 15:50:06.902340 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:06.902317    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-n2dql"]
Sep 10 15:50:06.954801 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:06.954773    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfrww\" (UniqueName: \"kubernetes.io/projected/ceb84345-001e-41f4-bcba-89ec0320dc6f-kube-api-access-nfrww\") pod \"authorino-59c974bd48-n2dql\" (UID: \"ceb84345-001e-41f4-bcba-89ec0320dc6f\") " pod="kuadrant-system/authorino-59c974bd48-n2dql"
Sep 10 15:50:07.055366 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:07.055333    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfrww\" (UniqueName: \"kubernetes.io/projected/ceb84345-001e-41f4-bcba-89ec0320dc6f-kube-api-access-nfrww\") pod \"authorino-59c974bd48-n2dql\" (UID: \"ceb84345-001e-41f4-bcba-89ec0320dc6f\") " pod="kuadrant-system/authorino-59c974bd48-n2dql"
Sep 10 15:50:07.064306 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:07.064277    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfrww\" (UniqueName: \"kubernetes.io/projected/ceb84345-001e-41f4-bcba-89ec0320dc6f-kube-api-access-nfrww\") pod \"authorino-59c974bd48-n2dql\" (UID: \"ceb84345-001e-41f4-bcba-89ec0320dc6f\") " pod="kuadrant-system/authorino-59c974bd48-n2dql"
Sep 10 15:50:07.206314 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:07.206286    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-n2dql"
Sep 10 15:50:07.477646 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:07.477571    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-n2dql"]
Sep 10 15:50:07.482014 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:50:07.481986    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podceb84345_001e_41f4_bcba_89ec0320dc6f.slice/crio-2678d1980e2796689fa44cbb2ca0286ce1bf4214f4bc9778eb22747cf9ec4009 WatchSource:0}: Error finding container 2678d1980e2796689fa44cbb2ca0286ce1bf4214f4bc9778eb22747cf9ec4009: Status 404 returned error can't find the container with id 2678d1980e2796689fa44cbb2ca0286ce1bf4214f4bc9778eb22747cf9ec4009
Sep 10 15:50:07.895125 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:07.895096    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-n2dql" event={"ID":"ceb84345-001e-41f4-bcba-89ec0320dc6f","Type":"ContainerStarted","Data":"2678d1980e2796689fa44cbb2ca0286ce1bf4214f4bc9778eb22747cf9ec4009"}
Sep 10 15:50:07.896560 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:07.896537    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-6j9zt" event={"ID":"06d00021-5706-4eb4-bdd8-50d30f5f1db1","Type":"ContainerStarted","Data":"b0b6e295b6564bef67a37c7bee48afa2ec449099cbd46a3a4d686efa85ba2701"}
Sep 10 15:50:07.914591 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:07.914545    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-6j9zt" podStartSLOduration=1.636658571 podStartE2EDuration="2.914530415s" totalImagesPullingTime="1.277871844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:07.912782192 +0000 UTC m=+486.717101464" watchObservedRunningTime="2026-09-10 15:50:07.914530415 +0000 UTC m=+486.718849676"
Sep 10 15:50:08.901694 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:08.901660    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-6j9zt"
Sep 10 15:50:09.908315 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:09.908282    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-n2dql" event={"ID":"ceb84345-001e-41f4-bcba-89ec0320dc6f","Type":"ContainerStarted","Data":"04e2f7ed3e04529cc28f44df3d5bf4baa6516383fa0ad8634faecd9cb2803ae0"}
Sep 10 15:50:09.925807 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:09.925760    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-n2dql" podStartSLOduration=2.236013404 podStartE2EDuration="3.925744803s" totalImagesPullingTime="1.689731399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:09.92404305 +0000 UTC m=+488.728362311" watchObservedRunningTime="2026-09-10 15:50:09.925744803 +0000 UTC m=+488.730064063"
Sep 10 15:50:19.909206 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:19.909152    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-6j9zt"
Sep 10 15:50:21.929130 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:21.929094    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6j9zt"]
Sep 10 15:50:21.929621 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:21.929304    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-6j9zt" podUID="06d00021-5706-4eb4-bdd8-50d30f5f1db1" containerName="limitador" containerID="cri-o://b0b6e295b6564bef67a37c7bee48afa2ec449099cbd46a3a4d686efa85ba2701" gracePeriod=30
Sep 10 15:50:22.876622 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:22.876594    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-6j9zt"
Sep 10 15:50:22.982316 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:22.982275    2564 generic.go:356] "Generic (PLEG): container finished" podID="06d00021-5706-4eb4-bdd8-50d30f5f1db1" containerID="b0b6e295b6564bef67a37c7bee48afa2ec449099cbd46a3a4d686efa85ba2701" exitCode=0
Sep 10 15:50:22.982316 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:22.982317    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-6j9zt" event={"ID":"06d00021-5706-4eb4-bdd8-50d30f5f1db1","Type":"ContainerDied","Data":"b0b6e295b6564bef67a37c7bee48afa2ec449099cbd46a3a4d686efa85ba2701"}
Sep 10 15:50:22.982316 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:22.982320    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-6j9zt"
Sep 10 15:50:22.982768 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:22.982358    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-6j9zt" event={"ID":"06d00021-5706-4eb4-bdd8-50d30f5f1db1","Type":"ContainerDied","Data":"e3d72c17cbae36ea4b305d08e4c59d302e019630b144c5e4ada460bd6a79d6c2"}
Sep 10 15:50:22.982768 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:22.982379    2564 scope.go:117] "RemoveContainer" containerID="b0b6e295b6564bef67a37c7bee48afa2ec449099cbd46a3a4d686efa85ba2701"
Sep 10 15:50:22.993136 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:22.993117    2564 scope.go:117] "RemoveContainer" containerID="b0b6e295b6564bef67a37c7bee48afa2ec449099cbd46a3a4d686efa85ba2701"
Sep 10 15:50:22.993526 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:50:22.993504    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0b6e295b6564bef67a37c7bee48afa2ec449099cbd46a3a4d686efa85ba2701\": container with ID starting with b0b6e295b6564bef67a37c7bee48afa2ec449099cbd46a3a4d686efa85ba2701 not found: ID does not exist" containerID="b0b6e295b6564bef67a37c7bee48afa2ec449099cbd46a3a4d686efa85ba2701"
Sep 10 15:50:22.993613 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:22.993544    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0b6e295b6564bef67a37c7bee48afa2ec449099cbd46a3a4d686efa85ba2701"} err="failed to get container status \"b0b6e295b6564bef67a37c7bee48afa2ec449099cbd46a3a4d686efa85ba2701\": rpc error: code = NotFound desc = could not find container \"b0b6e295b6564bef67a37c7bee48afa2ec449099cbd46a3a4d686efa85ba2701\": container with ID starting with b0b6e295b6564bef67a37c7bee48afa2ec449099cbd46a3a4d686efa85ba2701 not found: ID does not exist"
Sep 10 15:50:23.015610 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:23.015580    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9vc9\" (UniqueName: \"kubernetes.io/projected/06d00021-5706-4eb4-bdd8-50d30f5f1db1-kube-api-access-m9vc9\") pod \"06d00021-5706-4eb4-bdd8-50d30f5f1db1\" (UID: \"06d00021-5706-4eb4-bdd8-50d30f5f1db1\") "
Sep 10 15:50:23.015757 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:23.015633    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/06d00021-5706-4eb4-bdd8-50d30f5f1db1-config-file\") pod \"06d00021-5706-4eb4-bdd8-50d30f5f1db1\" (UID: \"06d00021-5706-4eb4-bdd8-50d30f5f1db1\") "
Sep 10 15:50:23.016018 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:23.015986    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d00021-5706-4eb4-bdd8-50d30f5f1db1-config-file" (OuterVolumeSpecName: "config-file") pod "06d00021-5706-4eb4-bdd8-50d30f5f1db1" (UID: "06d00021-5706-4eb4-bdd8-50d30f5f1db1"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:50:23.017859 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:23.017819    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d00021-5706-4eb4-bdd8-50d30f5f1db1-kube-api-access-m9vc9" (OuterVolumeSpecName: "kube-api-access-m9vc9") pod "06d00021-5706-4eb4-bdd8-50d30f5f1db1" (UID: "06d00021-5706-4eb4-bdd8-50d30f5f1db1"). InnerVolumeSpecName "kube-api-access-m9vc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:50:23.116785 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:23.116746    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9vc9\" (UniqueName: \"kubernetes.io/projected/06d00021-5706-4eb4-bdd8-50d30f5f1db1-kube-api-access-m9vc9\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:50:23.116785 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:23.116777    2564 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/06d00021-5706-4eb4-bdd8-50d30f5f1db1-config-file\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:50:23.304245 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:23.304212    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6j9zt"]
Sep 10 15:50:23.308147 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:23.308115    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-6j9zt"]
Sep 10 15:50:23.671932 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:23.671903    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06d00021-5706-4eb4-bdd8-50d30f5f1db1" path="/var/lib/kubelet/pods/06d00021-5706-4eb4-bdd8-50d30f5f1db1/volumes"
Sep 10 15:50:27.526882 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:27.526849    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-78kbv"]
Sep 10 15:50:27.527465 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:27.527448    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06d00021-5706-4eb4-bdd8-50d30f5f1db1" containerName="limitador"
Sep 10 15:50:27.527508 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:27.527468    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="06d00021-5706-4eb4-bdd8-50d30f5f1db1" containerName="limitador"
Sep 10 15:50:27.527576 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:27.527565    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="06d00021-5706-4eb4-bdd8-50d30f5f1db1" containerName="limitador"
Sep 10 15:50:27.530104 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:27.530086    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-78kbv"
Sep 10 15:50:27.533661 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:27.533637    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 10 15:50:27.533801 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:27.533637    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 15:50:27.534250 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:27.534185    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-2k6g9\""
Sep 10 15:50:27.534250 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:27.534203    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 15:50:27.537745 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:27.537723    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-78kbv"]
Sep 10 15:50:27.557796 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:27.557772    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftblv\" (UniqueName: \"kubernetes.io/projected/3869fc0f-cd42-4f52-8d4b-36a75af216d2-kube-api-access-ftblv\") pod \"postgres-69fb4594cf-78kbv\" (UID: \"3869fc0f-cd42-4f52-8d4b-36a75af216d2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-78kbv"
Sep 10 15:50:27.557899 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:27.557812    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3869fc0f-cd42-4f52-8d4b-36a75af216d2-data\") pod \"postgres-69fb4594cf-78kbv\" (UID: \"3869fc0f-cd42-4f52-8d4b-36a75af216d2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-78kbv"
Sep 10 15:50:27.658965 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:27.658927    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftblv\" (UniqueName: \"kubernetes.io/projected/3869fc0f-cd42-4f52-8d4b-36a75af216d2-kube-api-access-ftblv\") pod \"postgres-69fb4594cf-78kbv\" (UID: \"3869fc0f-cd42-4f52-8d4b-36a75af216d2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-78kbv"
Sep 10 15:50:27.659102 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:27.658987    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3869fc0f-cd42-4f52-8d4b-36a75af216d2-data\") pod \"postgres-69fb4594cf-78kbv\" (UID: \"3869fc0f-cd42-4f52-8d4b-36a75af216d2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-78kbv"
Sep 10 15:50:27.659393 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:27.659374    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3869fc0f-cd42-4f52-8d4b-36a75af216d2-data\") pod \"postgres-69fb4594cf-78kbv\" (UID: \"3869fc0f-cd42-4f52-8d4b-36a75af216d2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-78kbv"
Sep 10 15:50:27.670576 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:27.670546    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftblv\" (UniqueName: \"kubernetes.io/projected/3869fc0f-cd42-4f52-8d4b-36a75af216d2-kube-api-access-ftblv\") pod \"postgres-69fb4594cf-78kbv\" (UID: \"3869fc0f-cd42-4f52-8d4b-36a75af216d2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-78kbv"
Sep 10 15:50:27.843801 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:27.843706    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-78kbv"
Sep 10 15:50:28.123884 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:28.123861    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-78kbv"]
Sep 10 15:50:29.015556 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:29.015518    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-78kbv" event={"ID":"3869fc0f-cd42-4f52-8d4b-36a75af216d2","Type":"ContainerStarted","Data":"22dda5bd2e52f9bfdc089bf389ea7e258c43dfb65d904e1b280685938b64466b"}
Sep 10 15:50:48.115880 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:48.115843    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-78kbv" event={"ID":"3869fc0f-cd42-4f52-8d4b-36a75af216d2","Type":"ContainerStarted","Data":"6acf06dd81cd4815004e219e85c2db161ac89e373b194c23a74221b2023bde4c"}
Sep 10 15:50:48.116352 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:48.115966    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-78kbv"
Sep 10 15:50:48.134272 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:48.134196    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-78kbv" podStartSLOduration=1.684803139 podStartE2EDuration="21.134150375s" totalImagesPullingTime="19.449347236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:48.133454667 +0000 UTC m=+526.937773929" watchObservedRunningTime="2026-09-10 15:50:48.134150375 +0000 UTC m=+526.938469640"
Sep 10 15:50:54.150573 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:54.150493    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-78kbv"
Sep 10 15:50:54.943602 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:54.943559    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66886fc78-fg76n"]
Sep 10 15:50:54.946153 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:54.946135    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66886fc78-fg76n"
Sep 10 15:50:54.948393 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:54.948373    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 15:50:54.956109 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:54.956089    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66886fc78-fg76n"]
Sep 10 15:50:55.098340 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:55.098301    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2116085b-1414-4d03-8cb1-8e46afa82c44-tls-cert\") pod \"authorino-66886fc78-fg76n\" (UID: \"2116085b-1414-4d03-8cb1-8e46afa82c44\") " pod="kuadrant-system/authorino-66886fc78-fg76n"
Sep 10 15:50:55.098340 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:55.098345    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjprs\" (UniqueName: \"kubernetes.io/projected/2116085b-1414-4d03-8cb1-8e46afa82c44-kube-api-access-pjprs\") pod \"authorino-66886fc78-fg76n\" (UID: \"2116085b-1414-4d03-8cb1-8e46afa82c44\") " pod="kuadrant-system/authorino-66886fc78-fg76n"
Sep 10 15:50:55.199052 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:55.198956    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2116085b-1414-4d03-8cb1-8e46afa82c44-tls-cert\") pod \"authorino-66886fc78-fg76n\" (UID: \"2116085b-1414-4d03-8cb1-8e46afa82c44\") " pod="kuadrant-system/authorino-66886fc78-fg76n"
Sep 10 15:50:55.199052 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:55.199000    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjprs\" (UniqueName: \"kubernetes.io/projected/2116085b-1414-4d03-8cb1-8e46afa82c44-kube-api-access-pjprs\") pod \"authorino-66886fc78-fg76n\" (UID: \"2116085b-1414-4d03-8cb1-8e46afa82c44\") " pod="kuadrant-system/authorino-66886fc78-fg76n"
Sep 10 15:50:55.201520 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:55.201498    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2116085b-1414-4d03-8cb1-8e46afa82c44-tls-cert\") pod \"authorino-66886fc78-fg76n\" (UID: \"2116085b-1414-4d03-8cb1-8e46afa82c44\") " pod="kuadrant-system/authorino-66886fc78-fg76n"
Sep 10 15:50:55.214012 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:55.213990    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjprs\" (UniqueName: \"kubernetes.io/projected/2116085b-1414-4d03-8cb1-8e46afa82c44-kube-api-access-pjprs\") pod \"authorino-66886fc78-fg76n\" (UID: \"2116085b-1414-4d03-8cb1-8e46afa82c44\") " pod="kuadrant-system/authorino-66886fc78-fg76n"
Sep 10 15:50:55.256610 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:55.256581    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66886fc78-fg76n"
Sep 10 15:50:55.527511 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:55.527483    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66886fc78-fg76n"]
Sep 10 15:50:55.530029 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:50:55.530010    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2116085b_1414_4d03_8cb1_8e46afa82c44.slice/crio-d365203eb59c392dabb8f7a3becbde194d2420b166458aee33e7494ec0e2a9f1 WatchSource:0}: Error finding container d365203eb59c392dabb8f7a3becbde194d2420b166458aee33e7494ec0e2a9f1: Status 404 returned error can't find the container with id d365203eb59c392dabb8f7a3becbde194d2420b166458aee33e7494ec0e2a9f1
Sep 10 15:50:56.161816 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:56.161729    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66886fc78-fg76n" event={"ID":"2116085b-1414-4d03-8cb1-8e46afa82c44","Type":"ContainerStarted","Data":"d37ac7f8c72255e5575fe5aec0c9c6f5883c210e5325abc65355f197c3b6123e"}
Sep 10 15:50:56.161816 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:56.161769    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66886fc78-fg76n" event={"ID":"2116085b-1414-4d03-8cb1-8e46afa82c44","Type":"ContainerStarted","Data":"d365203eb59c392dabb8f7a3becbde194d2420b166458aee33e7494ec0e2a9f1"}
Sep 10 15:50:56.179360 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:56.179310    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-66886fc78-fg76n" podStartSLOduration=1.8281068249999999 podStartE2EDuration="2.179295288s" totalImagesPullingTime="351.188463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:56.177474491 +0000 UTC m=+534.981793763" watchObservedRunningTime="2026-09-10 15:50:56.179295288 +0000 UTC m=+534.983614549"
Sep 10 15:50:56.204030 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:56.203996    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-n2dql"]
Sep 10 15:50:56.204471 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:56.204205    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-n2dql" podUID="ceb84345-001e-41f4-bcba-89ec0320dc6f" containerName="authorino" containerID="cri-o://04e2f7ed3e04529cc28f44df3d5bf4baa6516383fa0ad8634faecd9cb2803ae0" gracePeriod=30
Sep 10 15:50:56.452944 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:56.452914    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-n2dql"
Sep 10 15:50:56.511102 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:56.511071    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfrww\" (UniqueName: \"kubernetes.io/projected/ceb84345-001e-41f4-bcba-89ec0320dc6f-kube-api-access-nfrww\") pod \"ceb84345-001e-41f4-bcba-89ec0320dc6f\" (UID: \"ceb84345-001e-41f4-bcba-89ec0320dc6f\") "
Sep 10 15:50:56.513307 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:56.513282    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb84345-001e-41f4-bcba-89ec0320dc6f-kube-api-access-nfrww" (OuterVolumeSpecName: "kube-api-access-nfrww") pod "ceb84345-001e-41f4-bcba-89ec0320dc6f" (UID: "ceb84345-001e-41f4-bcba-89ec0320dc6f"). InnerVolumeSpecName "kube-api-access-nfrww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:50:56.612450 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:56.612415    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfrww\" (UniqueName: \"kubernetes.io/projected/ceb84345-001e-41f4-bcba-89ec0320dc6f-kube-api-access-nfrww\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:50:57.168489 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:57.168450    2564 generic.go:356] "Generic (PLEG): container finished" podID="ceb84345-001e-41f4-bcba-89ec0320dc6f" containerID="04e2f7ed3e04529cc28f44df3d5bf4baa6516383fa0ad8634faecd9cb2803ae0" exitCode=0
Sep 10 15:50:57.168489 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:57.168492    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-n2dql"
Sep 10 15:50:57.168759 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:57.168532    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-n2dql" event={"ID":"ceb84345-001e-41f4-bcba-89ec0320dc6f","Type":"ContainerDied","Data":"04e2f7ed3e04529cc28f44df3d5bf4baa6516383fa0ad8634faecd9cb2803ae0"}
Sep 10 15:50:57.168759 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:57.168588    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-n2dql" event={"ID":"ceb84345-001e-41f4-bcba-89ec0320dc6f","Type":"ContainerDied","Data":"2678d1980e2796689fa44cbb2ca0286ce1bf4214f4bc9778eb22747cf9ec4009"}
Sep 10 15:50:57.168759 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:57.168621    2564 scope.go:117] "RemoveContainer" containerID="04e2f7ed3e04529cc28f44df3d5bf4baa6516383fa0ad8634faecd9cb2803ae0"
Sep 10 15:50:57.178332 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:57.178314    2564 scope.go:117] "RemoveContainer" containerID="04e2f7ed3e04529cc28f44df3d5bf4baa6516383fa0ad8634faecd9cb2803ae0"
Sep 10 15:50:57.178651 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:50:57.178632    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e2f7ed3e04529cc28f44df3d5bf4baa6516383fa0ad8634faecd9cb2803ae0\": container with ID starting with 04e2f7ed3e04529cc28f44df3d5bf4baa6516383fa0ad8634faecd9cb2803ae0 not found: ID does not exist" containerID="04e2f7ed3e04529cc28f44df3d5bf4baa6516383fa0ad8634faecd9cb2803ae0"
Sep 10 15:50:57.178722 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:57.178658    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e2f7ed3e04529cc28f44df3d5bf4baa6516383fa0ad8634faecd9cb2803ae0"} err="failed to get container status \"04e2f7ed3e04529cc28f44df3d5bf4baa6516383fa0ad8634faecd9cb2803ae0\": rpc error: code = NotFound desc = could not find container \"04e2f7ed3e04529cc28f44df3d5bf4baa6516383fa0ad8634faecd9cb2803ae0\": container with ID starting with 04e2f7ed3e04529cc28f44df3d5bf4baa6516383fa0ad8634faecd9cb2803ae0 not found: ID does not exist"
Sep 10 15:50:57.190490 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:57.190458    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-n2dql"]
Sep 10 15:50:57.194560 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:57.194535    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-n2dql"]
Sep 10 15:50:57.673727 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:50:57.673699    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceb84345-001e-41f4-bcba-89ec0320dc6f" path="/var/lib/kubelet/pods/ceb84345-001e-41f4-bcba-89ec0320dc6f/volumes"
Sep 10 15:51:00.887552 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:00.887516    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5774557b98-s4f69"]
Sep 10 15:51:00.888012 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:00.887997    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb84345-001e-41f4-bcba-89ec0320dc6f" containerName="authorino"
Sep 10 15:51:00.888012 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:00.888014    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb84345-001e-41f4-bcba-89ec0320dc6f" containerName="authorino"
Sep 10 15:51:00.888104 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:00.888071    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb84345-001e-41f4-bcba-89ec0320dc6f" containerName="authorino"
Sep 10 15:51:00.891191 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:00.891150    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5774557b98-s4f69"
Sep 10 15:51:00.893559 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:00.893530    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 15:51:00.893681 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:00.893556    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 15:51:00.893681 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:00.893587    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 15:51:00.893681 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:00.893656    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wf8fd\""
Sep 10 15:51:00.903065 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:00.903037    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5774557b98-s4f69"]
Sep 10 15:51:00.951895 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:00.951861    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl7f4\" (UniqueName: \"kubernetes.io/projected/53da8c3e-9974-4650-b491-54f97b69503c-kube-api-access-pl7f4\") pod \"maas-controller-5774557b98-s4f69\" (UID: \"53da8c3e-9974-4650-b491-54f97b69503c\") " pod="opendatahub/maas-controller-5774557b98-s4f69"
Sep 10 15:51:00.952062 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:00.951918    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53da8c3e-9974-4650-b491-54f97b69503c-webhook-cert\") pod \"maas-controller-5774557b98-s4f69\" (UID: \"53da8c3e-9974-4650-b491-54f97b69503c\") " pod="opendatahub/maas-controller-5774557b98-s4f69"
Sep 10 15:51:00.952062 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:00.951989    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53da8c3e-9974-4650-b491-54f97b69503c-metrics-certs\") pod \"maas-controller-5774557b98-s4f69\" (UID: \"53da8c3e-9974-4650-b491-54f97b69503c\") " pod="opendatahub/maas-controller-5774557b98-s4f69"
Sep 10 15:51:01.053566 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.053523    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl7f4\" (UniqueName: \"kubernetes.io/projected/53da8c3e-9974-4650-b491-54f97b69503c-kube-api-access-pl7f4\") pod \"maas-controller-5774557b98-s4f69\" (UID: \"53da8c3e-9974-4650-b491-54f97b69503c\") " pod="opendatahub/maas-controller-5774557b98-s4f69"
Sep 10 15:51:01.053793 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.053601    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53da8c3e-9974-4650-b491-54f97b69503c-webhook-cert\") pod \"maas-controller-5774557b98-s4f69\" (UID: \"53da8c3e-9974-4650-b491-54f97b69503c\") " pod="opendatahub/maas-controller-5774557b98-s4f69"
Sep 10 15:51:01.053793 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.053639    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53da8c3e-9974-4650-b491-54f97b69503c-metrics-certs\") pod \"maas-controller-5774557b98-s4f69\" (UID: \"53da8c3e-9974-4650-b491-54f97b69503c\") " pod="opendatahub/maas-controller-5774557b98-s4f69"
Sep 10 15:51:01.053913 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:51:01.053800    2564 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 10 15:51:01.053913 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:51:01.053870    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53da8c3e-9974-4650-b491-54f97b69503c-metrics-certs podName:53da8c3e-9974-4650-b491-54f97b69503c nodeName:}" failed. No retries permitted until 2026-09-10 15:51:01.553849947 +0000 UTC m=+540.358169186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53da8c3e-9974-4650-b491-54f97b69503c-metrics-certs") pod "maas-controller-5774557b98-s4f69" (UID: "53da8c3e-9974-4650-b491-54f97b69503c") : secret "maas-controller-metrics-tls" not found
Sep 10 15:51:01.056376 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.056354    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53da8c3e-9974-4650-b491-54f97b69503c-webhook-cert\") pod \"maas-controller-5774557b98-s4f69\" (UID: \"53da8c3e-9974-4650-b491-54f97b69503c\") " pod="opendatahub/maas-controller-5774557b98-s4f69"
Sep 10 15:51:01.065424 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.065398    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cd9db79db-rndpc"]
Sep 10 15:51:01.068359 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.068316    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd9db79db-rndpc"
Sep 10 15:51:01.069725 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.069702    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl7f4\" (UniqueName: \"kubernetes.io/projected/53da8c3e-9974-4650-b491-54f97b69503c-kube-api-access-pl7f4\") pod \"maas-controller-5774557b98-s4f69\" (UID: \"53da8c3e-9974-4650-b491-54f97b69503c\") " pod="opendatahub/maas-controller-5774557b98-s4f69"
Sep 10 15:51:01.078251 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.078222    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd9db79db-rndpc"]
Sep 10 15:51:01.154647 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.154548    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnhtr\" (UniqueName: \"kubernetes.io/projected/373a49de-5c83-48d4-a4d9-43d1b5ed6558-kube-api-access-bnhtr\") pod \"maas-controller-5cd9db79db-rndpc\" (UID: \"373a49de-5c83-48d4-a4d9-43d1b5ed6558\") " pod="opendatahub/maas-controller-5cd9db79db-rndpc"
Sep 10 15:51:01.154821 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.154649    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/373a49de-5c83-48d4-a4d9-43d1b5ed6558-webhook-cert\") pod \"maas-controller-5cd9db79db-rndpc\" (UID: \"373a49de-5c83-48d4-a4d9-43d1b5ed6558\") " pod="opendatahub/maas-controller-5cd9db79db-rndpc"
Sep 10 15:51:01.154821 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.154733    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373a49de-5c83-48d4-a4d9-43d1b5ed6558-metrics-certs\") pod \"maas-controller-5cd9db79db-rndpc\" (UID: \"373a49de-5c83-48d4-a4d9-43d1b5ed6558\") " pod="opendatahub/maas-controller-5cd9db79db-rndpc"
Sep 10 15:51:01.255431 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.255384    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/373a49de-5c83-48d4-a4d9-43d1b5ed6558-webhook-cert\") pod \"maas-controller-5cd9db79db-rndpc\" (UID: \"373a49de-5c83-48d4-a4d9-43d1b5ed6558\") " pod="opendatahub/maas-controller-5cd9db79db-rndpc"
Sep 10 15:51:01.255629 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.255452    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373a49de-5c83-48d4-a4d9-43d1b5ed6558-metrics-certs\") pod \"maas-controller-5cd9db79db-rndpc\" (UID: \"373a49de-5c83-48d4-a4d9-43d1b5ed6558\") " pod="opendatahub/maas-controller-5cd9db79db-rndpc"
Sep 10 15:51:01.255629 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.255508    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnhtr\" (UniqueName: \"kubernetes.io/projected/373a49de-5c83-48d4-a4d9-43d1b5ed6558-kube-api-access-bnhtr\") pod \"maas-controller-5cd9db79db-rndpc\" (UID: \"373a49de-5c83-48d4-a4d9-43d1b5ed6558\") " pod="opendatahub/maas-controller-5cd9db79db-rndpc"
Sep 10 15:51:01.258128 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.258098    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/373a49de-5c83-48d4-a4d9-43d1b5ed6558-webhook-cert\") pod \"maas-controller-5cd9db79db-rndpc\" (UID: \"373a49de-5c83-48d4-a4d9-43d1b5ed6558\") " pod="opendatahub/maas-controller-5cd9db79db-rndpc"
Sep 10 15:51:01.258254 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.258127    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373a49de-5c83-48d4-a4d9-43d1b5ed6558-metrics-certs\") pod \"maas-controller-5cd9db79db-rndpc\" (UID: \"373a49de-5c83-48d4-a4d9-43d1b5ed6558\") " pod="opendatahub/maas-controller-5cd9db79db-rndpc"
Sep 10 15:51:01.263896 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.263872    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnhtr\" (UniqueName: \"kubernetes.io/projected/373a49de-5c83-48d4-a4d9-43d1b5ed6558-kube-api-access-bnhtr\") pod \"maas-controller-5cd9db79db-rndpc\" (UID: \"373a49de-5c83-48d4-a4d9-43d1b5ed6558\") " pod="opendatahub/maas-controller-5cd9db79db-rndpc"
Sep 10 15:51:01.388718 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.388680    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd9db79db-rndpc"
Sep 10 15:51:01.558597 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.558559    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53da8c3e-9974-4650-b491-54f97b69503c-metrics-certs\") pod \"maas-controller-5774557b98-s4f69\" (UID: \"53da8c3e-9974-4650-b491-54f97b69503c\") " pod="opendatahub/maas-controller-5774557b98-s4f69"
Sep 10 15:51:01.561015 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.560996    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53da8c3e-9974-4650-b491-54f97b69503c-metrics-certs\") pod \"maas-controller-5774557b98-s4f69\" (UID: \"53da8c3e-9974-4650-b491-54f97b69503c\") " pod="opendatahub/maas-controller-5774557b98-s4f69"
Sep 10 15:51:01.659999 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.659975    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd9db79db-rndpc"]
Sep 10 15:51:01.661889 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:51:01.661862    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373a49de_5c83_48d4_a4d9_43d1b5ed6558.slice/crio-fbc6988ef8e414515c12ef3fcf15ba9ed1b6af85370b5e8cdd369ea9e1100065 WatchSource:0}: Error finding container fbc6988ef8e414515c12ef3fcf15ba9ed1b6af85370b5e8cdd369ea9e1100065: Status 404 returned error can't find the container with id fbc6988ef8e414515c12ef3fcf15ba9ed1b6af85370b5e8cdd369ea9e1100065
Sep 10 15:51:01.802475 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:01.802380    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5774557b98-s4f69"
Sep 10 15:51:02.076242 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:02.076217    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5774557b98-s4f69"]
Sep 10 15:51:02.078286 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:51:02.078259    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53da8c3e_9974_4650_b491_54f97b69503c.slice/crio-86a29cb791de28f202452f1125dd6b2d449b697b8e20ad8f866d686c48716b0c WatchSource:0}: Error finding container 86a29cb791de28f202452f1125dd6b2d449b697b8e20ad8f866d686c48716b0c: Status 404 returned error can't find the container with id 86a29cb791de28f202452f1125dd6b2d449b697b8e20ad8f866d686c48716b0c
Sep 10 15:51:02.197235 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:02.197200    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5774557b98-s4f69" event={"ID":"53da8c3e-9974-4650-b491-54f97b69503c","Type":"ContainerStarted","Data":"86a29cb791de28f202452f1125dd6b2d449b697b8e20ad8f866d686c48716b0c"}
Sep 10 15:51:02.198542 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:02.198519    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd9db79db-rndpc" event={"ID":"373a49de-5c83-48d4-a4d9-43d1b5ed6558","Type":"ContainerStarted","Data":"fbc6988ef8e414515c12ef3fcf15ba9ed1b6af85370b5e8cdd369ea9e1100065"}
Sep 10 15:51:04.242142 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:04.242116    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 15:51:05.218518 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:05.218468    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5774557b98-s4f69" event={"ID":"53da8c3e-9974-4650-b491-54f97b69503c","Type":"ContainerStarted","Data":"8e6009a4afe2a23aa6591c13167edd02ba522201575c477792dbf8c5902be6b7"}
Sep 10 15:51:05.218732 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:05.218546    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5774557b98-s4f69"
Sep 10 15:51:05.219999 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:05.219971    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd9db79db-rndpc" event={"ID":"373a49de-5c83-48d4-a4d9-43d1b5ed6558","Type":"ContainerStarted","Data":"d2d1b3b51aa8e4eaab44103a39052ca0487e36ddd31975baa7835cba2f9acc25"}
Sep 10 15:51:05.220079 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:05.220026    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cd9db79db-rndpc"
Sep 10 15:51:05.243690 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:05.243641    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5774557b98-s4f69" podStartSLOduration=3.076975738 podStartE2EDuration="5.24362865s" totalImagesPullingTime="2.166652912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:51:05.242209853 +0000 UTC m=+544.046529116" watchObservedRunningTime="2026-09-10 15:51:05.24362865 +0000 UTC m=+544.047947911"
Sep 10 15:51:05.266927 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:05.266880    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5cd9db79db-rndpc" podStartSLOduration=1.690907749 podStartE2EDuration="4.266863765s" totalImagesPullingTime="2.575956016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:51:05.264982864 +0000 UTC m=+544.069302129" watchObservedRunningTime="2026-09-10 15:51:05.266863765 +0000 UTC m=+544.071183027"
Sep 10 15:51:16.228569 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:16.228539    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5774557b98-s4f69"
Sep 10 15:51:16.229048 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:16.228744    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cd9db79db-rndpc"
Sep 10 15:51:16.297376 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:16.297340    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5774557b98-s4f69"]
Sep 10 15:51:16.297613 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:16.297584    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5774557b98-s4f69" podUID="53da8c3e-9974-4650-b491-54f97b69503c" containerName="manager" containerID="cri-o://8e6009a4afe2a23aa6591c13167edd02ba522201575c477792dbf8c5902be6b7" gracePeriod=30
Sep 10 15:51:16.546697 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:16.546674    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5774557b98-s4f69"
Sep 10 15:51:16.607088 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:16.607056    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53da8c3e-9974-4650-b491-54f97b69503c-webhook-cert\") pod \"53da8c3e-9974-4650-b491-54f97b69503c\" (UID: \"53da8c3e-9974-4650-b491-54f97b69503c\") "
Sep 10 15:51:16.607273 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:16.607107    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl7f4\" (UniqueName: \"kubernetes.io/projected/53da8c3e-9974-4650-b491-54f97b69503c-kube-api-access-pl7f4\") pod \"53da8c3e-9974-4650-b491-54f97b69503c\" (UID: \"53da8c3e-9974-4650-b491-54f97b69503c\") "
Sep 10 15:51:16.607273 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:16.607143    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53da8c3e-9974-4650-b491-54f97b69503c-metrics-certs\") pod \"53da8c3e-9974-4650-b491-54f97b69503c\" (UID: \"53da8c3e-9974-4650-b491-54f97b69503c\") "
Sep 10 15:51:16.609553 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:16.609507    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53da8c3e-9974-4650-b491-54f97b69503c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "53da8c3e-9974-4650-b491-54f97b69503c" (UID: "53da8c3e-9974-4650-b491-54f97b69503c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:51:16.609553 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:16.609519    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53da8c3e-9974-4650-b491-54f97b69503c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "53da8c3e-9974-4650-b491-54f97b69503c" (UID: "53da8c3e-9974-4650-b491-54f97b69503c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:51:16.609743 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:16.609576    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53da8c3e-9974-4650-b491-54f97b69503c-kube-api-access-pl7f4" (OuterVolumeSpecName: "kube-api-access-pl7f4") pod "53da8c3e-9974-4650-b491-54f97b69503c" (UID: "53da8c3e-9974-4650-b491-54f97b69503c"). InnerVolumeSpecName "kube-api-access-pl7f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:51:16.708359 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:16.708313    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53da8c3e-9974-4650-b491-54f97b69503c-webhook-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:51:16.708359 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:16.708351    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl7f4\" (UniqueName: \"kubernetes.io/projected/53da8c3e-9974-4650-b491-54f97b69503c-kube-api-access-pl7f4\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:51:16.708359 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:16.708363    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53da8c3e-9974-4650-b491-54f97b69503c-metrics-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:51:17.282999 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:17.282959    2564 generic.go:356] "Generic (PLEG): container finished" podID="53da8c3e-9974-4650-b491-54f97b69503c" containerID="8e6009a4afe2a23aa6591c13167edd02ba522201575c477792dbf8c5902be6b7" exitCode=0
Sep 10 15:51:17.283513 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:17.283022    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5774557b98-s4f69"
Sep 10 15:51:17.283513 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:17.283045    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5774557b98-s4f69" event={"ID":"53da8c3e-9974-4650-b491-54f97b69503c","Type":"ContainerDied","Data":"8e6009a4afe2a23aa6591c13167edd02ba522201575c477792dbf8c5902be6b7"}
Sep 10 15:51:17.283513 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:17.283129    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5774557b98-s4f69" event={"ID":"53da8c3e-9974-4650-b491-54f97b69503c","Type":"ContainerDied","Data":"86a29cb791de28f202452f1125dd6b2d449b697b8e20ad8f866d686c48716b0c"}
Sep 10 15:51:17.283513 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:17.283152    2564 scope.go:117] "RemoveContainer" containerID="8e6009a4afe2a23aa6591c13167edd02ba522201575c477792dbf8c5902be6b7"
Sep 10 15:51:17.293309 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:17.293289    2564 scope.go:117] "RemoveContainer" containerID="8e6009a4afe2a23aa6591c13167edd02ba522201575c477792dbf8c5902be6b7"
Sep 10 15:51:17.293596 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:51:17.293570    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e6009a4afe2a23aa6591c13167edd02ba522201575c477792dbf8c5902be6b7\": container with ID starting with 8e6009a4afe2a23aa6591c13167edd02ba522201575c477792dbf8c5902be6b7 not found: ID does not exist" containerID="8e6009a4afe2a23aa6591c13167edd02ba522201575c477792dbf8c5902be6b7"
Sep 10 15:51:17.293689 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:17.293607    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e6009a4afe2a23aa6591c13167edd02ba522201575c477792dbf8c5902be6b7"} err="failed to get container status \"8e6009a4afe2a23aa6591c13167edd02ba522201575c477792dbf8c5902be6b7\": rpc error: code = NotFound desc = could not find container \"8e6009a4afe2a23aa6591c13167edd02ba522201575c477792dbf8c5902be6b7\": container with ID starting with 8e6009a4afe2a23aa6591c13167edd02ba522201575c477792dbf8c5902be6b7 not found: ID does not exist"
Sep 10 15:51:17.308461 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:17.308433    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5774557b98-s4f69"]
Sep 10 15:51:17.311433 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:17.311410    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5774557b98-s4f69"]
Sep 10 15:51:17.327314 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:51:17.327279    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53da8c3e_9974_4650_b491_54f97b69503c.slice\": RecentStats: unable to find data in memory cache]"
Sep 10 15:51:17.672235 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:17.672204    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53da8c3e-9974-4650-b491-54f97b69503c" path="/var/lib/kubelet/pods/53da8c3e-9974-4650-b491-54f97b69503c/volumes"
Sep 10 15:51:38.669054 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.669012    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-dxfj9"]
Sep 10 15:51:38.669597 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.669522    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53da8c3e-9974-4650-b491-54f97b69503c" containerName="manager"
Sep 10 15:51:38.669597 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.669543    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="53da8c3e-9974-4650-b491-54f97b69503c" containerName="manager"
Sep 10 15:51:38.669671 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.669611    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="53da8c3e-9974-4650-b491-54f97b69503c" containerName="manager"
Sep 10 15:51:38.675695 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.675675    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxfj9"
Sep 10 15:51:38.678796 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.678770    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 15:51:38.680455 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.680434    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-4n4q2"]
Sep 10 15:51:38.683686 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.683630    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-4n4q2"
Sep 10 15:51:38.686020 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.685999    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-w7qrn\""
Sep 10 15:51:38.687186 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.687147    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-dxfj9"]
Sep 10 15:51:38.697419 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.697397    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-4n4q2"]
Sep 10 15:51:38.802886 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.802854    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xknw\" (UniqueName: \"kubernetes.io/projected/8dc6aa70-fbc7-48e4-bd55-2873ea032836-kube-api-access-5xknw\") pod \"payload-pre-processing-59fbc47bf8-dxfj9\" (UID: \"8dc6aa70-fbc7-48e4-bd55-2873ea032836\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxfj9"
Sep 10 15:51:38.803045 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.802902    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dc6aa70-fbc7-48e4-bd55-2873ea032836-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-dxfj9\" (UID: \"8dc6aa70-fbc7-48e4-bd55-2873ea032836\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxfj9"
Sep 10 15:51:38.803045 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.803008    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d551ac05-c043-4e13-90da-c8c978d709f7-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-4n4q2\" (UID: \"d551ac05-c043-4e13-90da-c8c978d709f7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4n4q2"
Sep 10 15:51:38.803147 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.803072    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj4v8\" (UniqueName: \"kubernetes.io/projected/d551ac05-c043-4e13-90da-c8c978d709f7-kube-api-access-xj4v8\") pod \"payload-processing-5b54b7fbb7-4n4q2\" (UID: \"d551ac05-c043-4e13-90da-c8c978d709f7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4n4q2"
Sep 10 15:51:38.904719 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.904682    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xknw\" (UniqueName: \"kubernetes.io/projected/8dc6aa70-fbc7-48e4-bd55-2873ea032836-kube-api-access-5xknw\") pod \"payload-pre-processing-59fbc47bf8-dxfj9\" (UID: \"8dc6aa70-fbc7-48e4-bd55-2873ea032836\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxfj9"
Sep 10 15:51:38.904925 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.904740    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dc6aa70-fbc7-48e4-bd55-2873ea032836-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-dxfj9\" (UID: \"8dc6aa70-fbc7-48e4-bd55-2873ea032836\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxfj9"
Sep 10 15:51:38.904925 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.904794    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d551ac05-c043-4e13-90da-c8c978d709f7-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-4n4q2\" (UID: \"d551ac05-c043-4e13-90da-c8c978d709f7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4n4q2"
Sep 10 15:51:38.904925 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.904826    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj4v8\" (UniqueName: \"kubernetes.io/projected/d551ac05-c043-4e13-90da-c8c978d709f7-kube-api-access-xj4v8\") pod \"payload-processing-5b54b7fbb7-4n4q2\" (UID: \"d551ac05-c043-4e13-90da-c8c978d709f7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4n4q2"
Sep 10 15:51:38.905918 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.905879    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dc6aa70-fbc7-48e4-bd55-2873ea032836-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-dxfj9\" (UID: \"8dc6aa70-fbc7-48e4-bd55-2873ea032836\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxfj9"
Sep 10 15:51:38.906056 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.905919    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d551ac05-c043-4e13-90da-c8c978d709f7-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-4n4q2\" (UID: \"d551ac05-c043-4e13-90da-c8c978d709f7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4n4q2"
Sep 10 15:51:38.913528 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.913476    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xknw\" (UniqueName: \"kubernetes.io/projected/8dc6aa70-fbc7-48e4-bd55-2873ea032836-kube-api-access-5xknw\") pod \"payload-pre-processing-59fbc47bf8-dxfj9\" (UID: \"8dc6aa70-fbc7-48e4-bd55-2873ea032836\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxfj9"
Sep 10 15:51:38.913770 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.913743    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj4v8\" (UniqueName: \"kubernetes.io/projected/d551ac05-c043-4e13-90da-c8c978d709f7-kube-api-access-xj4v8\") pod \"payload-processing-5b54b7fbb7-4n4q2\" (UID: \"d551ac05-c043-4e13-90da-c8c978d709f7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4n4q2"
Sep 10 15:51:38.987900 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:38.987807    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxfj9"
Sep 10 15:51:39.000609 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:39.000582    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-4n4q2"
Sep 10 15:51:39.265846 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:39.265769    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-dxfj9"]
Sep 10 15:51:39.269768 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:51:39.269728    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc6aa70_fbc7_48e4_bd55_2873ea032836.slice/crio-219e07370b926643a637bcc673186374b3488f41c51e3802f61d01592439069b WatchSource:0}: Error finding container 219e07370b926643a637bcc673186374b3488f41c51e3802f61d01592439069b: Status 404 returned error can't find the container with id 219e07370b926643a637bcc673186374b3488f41c51e3802f61d01592439069b
Sep 10 15:51:39.290765 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:39.290674    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-4n4q2"]
Sep 10 15:51:39.292935 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:51:39.292903    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd551ac05_c043_4e13_90da_c8c978d709f7.slice/crio-8c53dbd02fd712edac5e00c2df97e3654efec9fb85700975959061b7696fad72 WatchSource:0}: Error finding container 8c53dbd02fd712edac5e00c2df97e3654efec9fb85700975959061b7696fad72: Status 404 returned error can't find the container with id 8c53dbd02fd712edac5e00c2df97e3654efec9fb85700975959061b7696fad72
Sep 10 15:51:39.404557 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:39.404513    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-4n4q2" event={"ID":"d551ac05-c043-4e13-90da-c8c978d709f7","Type":"ContainerStarted","Data":"8c53dbd02fd712edac5e00c2df97e3654efec9fb85700975959061b7696fad72"}
Sep 10 15:51:39.405884 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:39.405856    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxfj9" event={"ID":"8dc6aa70-fbc7-48e4-bd55-2873ea032836","Type":"ContainerStarted","Data":"219e07370b926643a637bcc673186374b3488f41c51e3802f61d01592439069b"}
Sep 10 15:51:42.442239 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:42.442203    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-4n4q2" event={"ID":"d551ac05-c043-4e13-90da-c8c978d709f7","Type":"ContainerStarted","Data":"cc6545cc3e7e0622a3d528c0a0238d23163bae51b082c3e69c62dc5c9d4092aa"}
Sep 10 15:51:42.442670 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:42.442333    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-4n4q2"
Sep 10 15:51:42.443829 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:42.443803    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxfj9" event={"ID":"8dc6aa70-fbc7-48e4-bd55-2873ea032836","Type":"ContainerStarted","Data":"a85e50d97ca4e40e8829deb3ea74f81279f9529b4005fa0e5643a7868aab4501"}
Sep 10 15:51:42.443921 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:42.443858    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxfj9"
Sep 10 15:51:42.463234 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:42.463187    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-4n4q2" podStartSLOduration=2.266492982 podStartE2EDuration="4.463176052s" totalImagesPullingTime="2.19668307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:51:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:51:42.460736731 +0000 UTC m=+581.265055992" watchObservedRunningTime="2026-09-10 15:51:42.463176052 +0000 UTC m=+581.267495306"
Sep 10 15:51:42.479391 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:42.479348    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxfj9" podStartSLOduration=2.270309429 podStartE2EDuration="4.479335429s" totalImagesPullingTime="2.209026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:51:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:51:42.47804003 +0000 UTC m=+581.282359292" watchObservedRunningTime="2026-09-10 15:51:42.479335429 +0000 UTC m=+581.283654690"
Sep 10 15:51:48.482555 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:48.482472    2564 generic.go:356] "Generic (PLEG): container finished" podID="5d6bce58-c1f0-46be-8110-c84cab62ff75" containerID="daba2ded886f423927ef0759b990728fb15b16999145a973ffa934a0eca01907" exitCode=0
Sep 10 15:51:48.483045 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:48.482556    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz" event={"ID":"5d6bce58-c1f0-46be-8110-c84cab62ff75","Type":"ContainerDied","Data":"daba2ded886f423927ef0759b990728fb15b16999145a973ffa934a0eca01907"}
Sep 10 15:51:48.483109 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:48.483075    2564 scope.go:117] "RemoveContainer" containerID="daba2ded886f423927ef0759b990728fb15b16999145a973ffa934a0eca01907"
Sep 10 15:51:48.484130 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:48.484101    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:51:48.484245 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:48.484196    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:51:48.484245 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:48.484228    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:51:49.489610 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:49.489573    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz" event={"ID":"5d6bce58-c1f0-46be-8110-c84cab62ff75","Type":"ContainerStarted","Data":"31444125f7b76a9c87bbb8133ffaa7ce81a1ba52a3edd0a3a2b5c90c1a4796be"}
Sep 10 15:51:50.417107 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:50.417066    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:51:50.418330 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:50.418298    2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-wgwhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 10 15:51:50.418552 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:50.418526    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz" podUID="5d6bce58-c1f0-46be-8110-c84cab62ff75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 10 15:51:50.883614 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:50.883582    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:51:51.417729 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:51.417243    2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-wgwhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 10 15:51:51.417729 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:51.417328    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz" podUID="5d6bce58-c1f0-46be-8110-c84cab62ff75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 10 15:51:52.416879 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:52.416832    2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-wgwhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 10 15:51:52.417393 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:52.416914    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz" podUID="5d6bce58-c1f0-46be-8110-c84cab62ff75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 10 15:51:53.416528 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:53.416484    2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-wgwhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 10 15:51:53.416744 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:53.416561    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz" podUID="5d6bce58-c1f0-46be-8110-c84cab62ff75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 10 15:51:53.450867 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:53.450838    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-4n4q2"
Sep 10 15:51:53.451411 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:53.451134    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dxfj9"
Sep 10 15:51:54.471254 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:54.471213    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:51:54.472455 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:51:54.472429    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wgwhz"
Sep 10 15:52:02.295300 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:02.295271    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-snm4z_f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f/console-operator/1.log"
Sep 10 15:52:02.296132 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:02.296101    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-snm4z_f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f/console-operator/1.log"
Sep 10 15:52:02.308233 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:02.308209    2564 scope.go:117] "RemoveContainer" containerID="9a2cd9f3c7ef6434c030dbc566482b92f2fa2110132d27a63c88e8aa995f0209"
Sep 10 15:52:16.758909 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.758867    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"]
Sep 10 15:52:16.764678 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.764656    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:16.768132 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.768105    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-h87hn\""
Sep 10 15:52:16.768132 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.768146    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 15:52:16.768367 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.768239    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 10 15:52:16.768367 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.768239    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 15:52:16.771585 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.771563    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"]
Sep 10 15:52:16.870583 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.870544    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4\" (UID: \"204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:16.870583 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.870581    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hh6g\" (UniqueName: \"kubernetes.io/projected/204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9-kube-api-access-8hh6g\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4\" (UID: \"204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:16.870836 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.870613    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4\" (UID: \"204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:16.870836 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.870710    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4\" (UID: \"204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:16.870836 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.870743    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4\" (UID: \"204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:16.870836 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.870776    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4\" (UID: \"204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:16.972174 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.972129    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4\" (UID: \"204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:16.972377 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.972194    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hh6g\" (UniqueName: \"kubernetes.io/projected/204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9-kube-api-access-8hh6g\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4\" (UID: \"204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:16.972377 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.972250    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4\" (UID: \"204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:16.972377 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.972282    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4\" (UID: \"204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:16.972377 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.972311    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4\" (UID: \"204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:16.972377 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.972352    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4\" (UID: \"204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:16.972835 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.972799    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4\" (UID: \"204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:16.972976 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.972847    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4\" (UID: \"204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:16.972976 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.972884    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4\" (UID: \"204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:16.974775 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.974745    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4\" (UID: \"204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:16.975071 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.975052    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4\" (UID: \"204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:16.980658 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:16.980636    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hh6g\" (UniqueName: \"kubernetes.io/projected/204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9-kube-api-access-8hh6g\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4\" (UID: \"204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:17.075941 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:17.075852    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:17.354637 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:17.354614    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"]
Sep 10 15:52:17.357638 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:52:17.357613    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204f1b8f_4b46_4c63_a48a_8f9a81d5f0a9.slice/crio-f87813d335bb5e510e8317f9c140ae038fa44e09f706ed5b7efe47eb7d1d7f6b WatchSource:0}: Error finding container f87813d335bb5e510e8317f9c140ae038fa44e09f706ed5b7efe47eb7d1d7f6b: Status 404 returned error can't find the container with id f87813d335bb5e510e8317f9c140ae038fa44e09f706ed5b7efe47eb7d1d7f6b
Sep 10 15:52:17.659446 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:17.659356    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4" event={"ID":"204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9","Type":"ContainerStarted","Data":"f87813d335bb5e510e8317f9c140ae038fa44e09f706ed5b7efe47eb7d1d7f6b"}
Sep 10 15:52:18.668578 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:18.668547    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4" event={"ID":"204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9","Type":"ContainerStarted","Data":"d4d7a57050d74cbadbf3a4390f6d85cfe8cefae03c38de34832eb93d34fe9f33"}
Sep 10 15:52:18.691965 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:18.691852    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4" podStartSLOduration=1.620600467 podStartE2EDuration="2.691832024s" totalImagesPullingTime="1.071231557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:52:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:52:18.6903607 +0000 UTC m=+617.494679964" watchObservedRunningTime="2026-09-10 15:52:18.691832024 +0000 UTC m=+617.496151286"
Sep 10 15:52:21.952848 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:21.952809    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"]
Sep 10 15:52:21.957566 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:21.957536    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:21.960359 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:21.960336    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 10 15:52:21.969376 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:21.969348    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"]
Sep 10 15:52:22.022708 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.022671    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a29fdc8-8361-4a32-9ce4-765dfab4f6cd-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-grqxg\" (UID: \"3a29fdc8-8361-4a32-9ce4-765dfab4f6cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:22.022708 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.022712    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a29fdc8-8361-4a32-9ce4-765dfab4f6cd-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-grqxg\" (UID: \"3a29fdc8-8361-4a32-9ce4-765dfab4f6cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:22.022970 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.022749    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a29fdc8-8361-4a32-9ce4-765dfab4f6cd-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-grqxg\" (UID: \"3a29fdc8-8361-4a32-9ce4-765dfab4f6cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:22.022970 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.022766    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a29fdc8-8361-4a32-9ce4-765dfab4f6cd-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-grqxg\" (UID: \"3a29fdc8-8361-4a32-9ce4-765dfab4f6cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:22.022970 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.022867    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q72nx\" (UniqueName: \"kubernetes.io/projected/3a29fdc8-8361-4a32-9ce4-765dfab4f6cd-kube-api-access-q72nx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-grqxg\" (UID: \"3a29fdc8-8361-4a32-9ce4-765dfab4f6cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:22.022970 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.022946    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a29fdc8-8361-4a32-9ce4-765dfab4f6cd-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-grqxg\" (UID: \"3a29fdc8-8361-4a32-9ce4-765dfab4f6cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:22.124051 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.124011    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a29fdc8-8361-4a32-9ce4-765dfab4f6cd-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-grqxg\" (UID: \"3a29fdc8-8361-4a32-9ce4-765dfab4f6cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:22.124051 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.124057    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a29fdc8-8361-4a32-9ce4-765dfab4f6cd-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-grqxg\" (UID: \"3a29fdc8-8361-4a32-9ce4-765dfab4f6cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:22.124321 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.124089    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a29fdc8-8361-4a32-9ce4-765dfab4f6cd-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-grqxg\" (UID: \"3a29fdc8-8361-4a32-9ce4-765dfab4f6cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:22.124321 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.124137    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a29fdc8-8361-4a32-9ce4-765dfab4f6cd-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-grqxg\" (UID: \"3a29fdc8-8361-4a32-9ce4-765dfab4f6cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:22.124321 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.124191    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a29fdc8-8361-4a32-9ce4-765dfab4f6cd-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-grqxg\" (UID: \"3a29fdc8-8361-4a32-9ce4-765dfab4f6cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:22.124321 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.124269    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q72nx\" (UniqueName: \"kubernetes.io/projected/3a29fdc8-8361-4a32-9ce4-765dfab4f6cd-kube-api-access-q72nx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-grqxg\" (UID: \"3a29fdc8-8361-4a32-9ce4-765dfab4f6cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:22.124532 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.124498    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a29fdc8-8361-4a32-9ce4-765dfab4f6cd-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-grqxg\" (UID: \"3a29fdc8-8361-4a32-9ce4-765dfab4f6cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:22.124584 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.124548    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a29fdc8-8361-4a32-9ce4-765dfab4f6cd-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-grqxg\" (UID: \"3a29fdc8-8361-4a32-9ce4-765dfab4f6cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:22.124670 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.124646    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a29fdc8-8361-4a32-9ce4-765dfab4f6cd-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-grqxg\" (UID: \"3a29fdc8-8361-4a32-9ce4-765dfab4f6cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:22.126670 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.126648    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a29fdc8-8361-4a32-9ce4-765dfab4f6cd-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-grqxg\" (UID: \"3a29fdc8-8361-4a32-9ce4-765dfab4f6cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:22.126942 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.126879    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a29fdc8-8361-4a32-9ce4-765dfab4f6cd-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-grqxg\" (UID: \"3a29fdc8-8361-4a32-9ce4-765dfab4f6cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:22.134255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.134223    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q72nx\" (UniqueName: \"kubernetes.io/projected/3a29fdc8-8361-4a32-9ce4-765dfab4f6cd-kube-api-access-q72nx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-grqxg\" (UID: \"3a29fdc8-8361-4a32-9ce4-765dfab4f6cd\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:22.270986 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.270907    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:22.545604 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.545581    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"]
Sep 10 15:52:22.547543 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:52:22.547517    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a29fdc8_8361_4a32_9ce4_765dfab4f6cd.slice/crio-15c472289a5107e1b916a8930cfc899c5fa0e6d87a3621aca86555ec1d3eae3d WatchSource:0}: Error finding container 15c472289a5107e1b916a8930cfc899c5fa0e6d87a3621aca86555ec1d3eae3d: Status 404 returned error can't find the container with id 15c472289a5107e1b916a8930cfc899c5fa0e6d87a3621aca86555ec1d3eae3d
Sep 10 15:52:22.548932 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.548914    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:52:22.697802 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:22.695554    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg" event={"ID":"3a29fdc8-8361-4a32-9ce4-765dfab4f6cd","Type":"ContainerStarted","Data":"15c472289a5107e1b916a8930cfc899c5fa0e6d87a3621aca86555ec1d3eae3d"}
Sep 10 15:52:23.705298 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:23.705254    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg" event={"ID":"3a29fdc8-8361-4a32-9ce4-765dfab4f6cd","Type":"ContainerStarted","Data":"a68b1a792af391e30dc96687ef254f2f3dbc04d36f47cdda0db9686d51732fe8"}
Sep 10 15:52:23.723650 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:23.723589    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg" podStartSLOduration=2.489994523 podStartE2EDuration="2.723567797s" totalImagesPullingTime="233.573274ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:52:23.723451678 +0000 UTC m=+622.527770946" watchObservedRunningTime="2026-09-10 15:52:23.723567797 +0000 UTC m=+622.527887060"
Sep 10 15:52:27.076037 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:27.075995    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:27.076496 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:27.076070    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:27.089052 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:27.089028    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:27.738333 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:27.738295    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4"
Sep 10 15:52:32.271105 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:32.271066    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:32.271704 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:32.271148    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:32.283567 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:32.283544    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:52:32.765037 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:52:32.765009    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-grqxg"
Sep 10 15:54:07.841542 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:07.841449    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66886fc78-fg76n"]
Sep 10 15:54:07.842095 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:07.841712    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-66886fc78-fg76n" podUID="2116085b-1414-4d03-8cb1-8e46afa82c44" containerName="authorino" containerID="cri-o://d37ac7f8c72255e5575fe5aec0c9c6f5883c210e5325abc65355f197c3b6123e" gracePeriod=30
Sep 10 15:54:08.106560 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:08.106500    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66886fc78-fg76n"
Sep 10 15:54:08.142844 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:08.142815    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2116085b-1414-4d03-8cb1-8e46afa82c44-tls-cert\") pod \"2116085b-1414-4d03-8cb1-8e46afa82c44\" (UID: \"2116085b-1414-4d03-8cb1-8e46afa82c44\") "
Sep 10 15:54:08.143024 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:08.142853    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjprs\" (UniqueName: \"kubernetes.io/projected/2116085b-1414-4d03-8cb1-8e46afa82c44-kube-api-access-pjprs\") pod \"2116085b-1414-4d03-8cb1-8e46afa82c44\" (UID: \"2116085b-1414-4d03-8cb1-8e46afa82c44\") "
Sep 10 15:54:08.145271 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:08.145239    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2116085b-1414-4d03-8cb1-8e46afa82c44-kube-api-access-pjprs" (OuterVolumeSpecName: "kube-api-access-pjprs") pod "2116085b-1414-4d03-8cb1-8e46afa82c44" (UID: "2116085b-1414-4d03-8cb1-8e46afa82c44"). InnerVolumeSpecName "kube-api-access-pjprs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:54:08.154412 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:08.154382    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2116085b-1414-4d03-8cb1-8e46afa82c44-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "2116085b-1414-4d03-8cb1-8e46afa82c44" (UID: "2116085b-1414-4d03-8cb1-8e46afa82c44"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:54:08.243796 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:08.243762    2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2116085b-1414-4d03-8cb1-8e46afa82c44-tls-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:54:08.243796 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:08.243792    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjprs\" (UniqueName: \"kubernetes.io/projected/2116085b-1414-4d03-8cb1-8e46afa82c44-kube-api-access-pjprs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:54:08.249268 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:08.249239    2564 generic.go:356] "Generic (PLEG): container finished" podID="2116085b-1414-4d03-8cb1-8e46afa82c44" containerID="d37ac7f8c72255e5575fe5aec0c9c6f5883c210e5325abc65355f197c3b6123e" exitCode=0
Sep 10 15:54:08.249424 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:08.249285    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66886fc78-fg76n"
Sep 10 15:54:08.249424 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:08.249301    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66886fc78-fg76n" event={"ID":"2116085b-1414-4d03-8cb1-8e46afa82c44","Type":"ContainerDied","Data":"d37ac7f8c72255e5575fe5aec0c9c6f5883c210e5325abc65355f197c3b6123e"}
Sep 10 15:54:08.249424 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:08.249333    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66886fc78-fg76n" event={"ID":"2116085b-1414-4d03-8cb1-8e46afa82c44","Type":"ContainerDied","Data":"d365203eb59c392dabb8f7a3becbde194d2420b166458aee33e7494ec0e2a9f1"}
Sep 10 15:54:08.249424 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:08.249350    2564 scope.go:117] "RemoveContainer" containerID="d37ac7f8c72255e5575fe5aec0c9c6f5883c210e5325abc65355f197c3b6123e"
Sep 10 15:54:08.259198 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:08.259174    2564 scope.go:117] "RemoveContainer" containerID="d37ac7f8c72255e5575fe5aec0c9c6f5883c210e5325abc65355f197c3b6123e"
Sep 10 15:54:08.259468 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:54:08.259448    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37ac7f8c72255e5575fe5aec0c9c6f5883c210e5325abc65355f197c3b6123e\": container with ID starting with d37ac7f8c72255e5575fe5aec0c9c6f5883c210e5325abc65355f197c3b6123e not found: ID does not exist" containerID="d37ac7f8c72255e5575fe5aec0c9c6f5883c210e5325abc65355f197c3b6123e"
Sep 10 15:54:08.259536 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:08.259483    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37ac7f8c72255e5575fe5aec0c9c6f5883c210e5325abc65355f197c3b6123e"} err="failed to get container status \"d37ac7f8c72255e5575fe5aec0c9c6f5883c210e5325abc65355f197c3b6123e\": rpc error: code = NotFound desc = could not find container \"d37ac7f8c72255e5575fe5aec0c9c6f5883c210e5325abc65355f197c3b6123e\": container with ID starting with d37ac7f8c72255e5575fe5aec0c9c6f5883c210e5325abc65355f197c3b6123e not found: ID does not exist"
Sep 10 15:54:08.275057 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:08.275031    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66886fc78-fg76n"]
Sep 10 15:54:08.279356 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:08.279332    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-66886fc78-fg76n"]
Sep 10 15:54:09.673116 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:09.673081    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2116085b-1414-4d03-8cb1-8e46afa82c44" path="/var/lib/kubelet/pods/2116085b-1414-4d03-8cb1-8e46afa82c44/volumes"
Sep 10 15:54:27.703882 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.703843    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"]
Sep 10 15:54:27.704528 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.704506    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2116085b-1414-4d03-8cb1-8e46afa82c44" containerName="authorino"
Sep 10 15:54:27.704572 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.704534    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2116085b-1414-4d03-8cb1-8e46afa82c44" containerName="authorino"
Sep 10 15:54:27.704656 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.704643    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2116085b-1414-4d03-8cb1-8e46afa82c44" containerName="authorino"
Sep 10 15:54:27.707987 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.707961    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.710481 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.710462    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-83dcaf-openshift-default-dockercfg-9gf96\""
Sep 10 15:54:27.722671 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.722586    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"]
Sep 10 15:54:27.822359 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.822321    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-istio-envoy\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.822554 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.822365    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3765fb24-410e-4107-856c-af41b073836b-istiod-ca-cert\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.822554 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.822395    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-workload-certs\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.822554 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.822504    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3765fb24-410e-4107-856c-af41b073836b-istio-podinfo\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.822554 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.822547    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3765fb24-410e-4107-856c-af41b073836b-istio-token\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.822738 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.822641    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-credential-socket\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.822738 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.822679    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-istio-data\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.822808 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.822750    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9lr8\" (UniqueName: \"kubernetes.io/projected/3765fb24-410e-4107-856c-af41b073836b-kube-api-access-t9lr8\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.822808 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.822795    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-workload-socket\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.923560 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.923523    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3765fb24-410e-4107-856c-af41b073836b-istio-podinfo\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.923733 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.923579    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3765fb24-410e-4107-856c-af41b073836b-istio-token\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.923852 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.923827    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-credential-socket\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.923905 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.923892    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-istio-data\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.923951 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.923940    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9lr8\" (UniqueName: \"kubernetes.io/projected/3765fb24-410e-4107-856c-af41b073836b-kube-api-access-t9lr8\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.923998 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.923973    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-workload-socket\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.924055 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.924005    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-istio-envoy\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.924055 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.924035    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3765fb24-410e-4107-856c-af41b073836b-istiod-ca-cert\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.924243 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.924217    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-workload-certs\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.924335 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.924240    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-credential-socket\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.924335 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.924279    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-istio-data\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.924448 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.924357    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-workload-socket\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.924598 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.924575    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-workload-certs\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.924866 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.924846    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3765fb24-410e-4107-856c-af41b073836b-istiod-ca-cert\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.926457 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.926427    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3765fb24-410e-4107-856c-af41b073836b-istio-podinfo\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.926546 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.926489    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-istio-envoy\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.931819 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.931796    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3765fb24-410e-4107-856c-af41b073836b-istio-token\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:27.932141 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:27.932117    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9lr8\" (UniqueName: \"kubernetes.io/projected/3765fb24-410e-4107-856c-af41b073836b-kube-api-access-t9lr8\") pod \"e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") " pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:28.019348 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:28.019263    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:28.299630 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:28.299597    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"]
Sep 10 15:54:28.301817 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:54:28.301786    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3765fb24_410e_4107_856c_af41b073836b.slice/crio-26644aa2b71e2341108cd7f5e93fe1d66d6d0e3a1163eb909fc4d7ae9c3bbd2b WatchSource:0}: Error finding container 26644aa2b71e2341108cd7f5e93fe1d66d6d0e3a1163eb909fc4d7ae9c3bbd2b: Status 404 returned error can't find the container with id 26644aa2b71e2341108cd7f5e93fe1d66d6d0e3a1163eb909fc4d7ae9c3bbd2b
Sep 10 15:54:28.304370 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:28.304338    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:54:28.304467 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:28.304407    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:54:28.304467 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:28.304440    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:54:28.362523 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:28.362358    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg" event={"ID":"3765fb24-410e-4107-856c-af41b073836b","Type":"ContainerStarted","Data":"26644aa2b71e2341108cd7f5e93fe1d66d6d0e3a1163eb909fc4d7ae9c3bbd2b"}
Sep 10 15:54:29.369412 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:29.369370    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg" event={"ID":"3765fb24-410e-4107-856c-af41b073836b","Type":"ContainerStarted","Data":"f04919c4dfd1a5a3ad150812cc7b6f28cc1f16a01dbbb5ae076410c8f63527c6"}
Sep 10 15:54:29.389466 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:29.389413    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg" podStartSLOduration=2.389396218 podStartE2EDuration="2.389396218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:29.388197298 +0000 UTC m=+748.192516552" watchObservedRunningTime="2026-09-10 15:54:29.389396218 +0000 UTC m=+748.193715479"
Sep 10 15:54:30.020175 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:30.020122    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:30.025927 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:30.025898    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:30.374567 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:30.374479    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:30.375538 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:30.375519    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:33.508442 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:33.508409    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v"]
Sep 10 15:54:33.512346 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:33.512328    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v"
Sep 10 15:54:33.514929 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:33.514907    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-83dcaf\""
Sep 10 15:54:33.515116 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:33.514974    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-83dcaf-dockercfg-q7dnf\""
Sep 10 15:54:33.522534 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:33.522511    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v"]
Sep 10 15:54:33.580643 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:33.580608    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da45bba5-1d5b-4a1c-8ff2-61bead26ab19-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v\" (UID: \"da45bba5-1d5b-4a1c-8ff2-61bead26ab19\") " pod="openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v"
Sep 10 15:54:33.580828 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:33.580776    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25llz\" (UniqueName: \"kubernetes.io/projected/da45bba5-1d5b-4a1c-8ff2-61bead26ab19-kube-api-access-25llz\") pod \"payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v\" (UID: \"da45bba5-1d5b-4a1c-8ff2-61bead26ab19\") " pod="openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v"
Sep 10 15:54:33.681752 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:33.681717    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da45bba5-1d5b-4a1c-8ff2-61bead26ab19-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v\" (UID: \"da45bba5-1d5b-4a1c-8ff2-61bead26ab19\") " pod="openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v"
Sep 10 15:54:33.681950 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:33.681879    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25llz\" (UniqueName: \"kubernetes.io/projected/da45bba5-1d5b-4a1c-8ff2-61bead26ab19-kube-api-access-25llz\") pod \"payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v\" (UID: \"da45bba5-1d5b-4a1c-8ff2-61bead26ab19\") " pod="openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v"
Sep 10 15:54:33.682724 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:33.682701    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da45bba5-1d5b-4a1c-8ff2-61bead26ab19-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v\" (UID: \"da45bba5-1d5b-4a1c-8ff2-61bead26ab19\") " pod="openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v"
Sep 10 15:54:33.692787 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:33.692756    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25llz\" (UniqueName: \"kubernetes.io/projected/da45bba5-1d5b-4a1c-8ff2-61bead26ab19-kube-api-access-25llz\") pod \"payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v\" (UID: \"da45bba5-1d5b-4a1c-8ff2-61bead26ab19\") " pod="openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v"
Sep 10 15:54:33.824982 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:33.824886    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v"
Sep 10 15:54:34.106871 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:34.106792    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v"]
Sep 10 15:54:34.109924 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:54:34.109901    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda45bba5_1d5b_4a1c_8ff2_61bead26ab19.slice/crio-13a78dfe1c1bc11da131a4d9ade124a71dc6e268049539a47837e51c3b92b91e WatchSource:0}: Error finding container 13a78dfe1c1bc11da131a4d9ade124a71dc6e268049539a47837e51c3b92b91e: Status 404 returned error can't find the container with id 13a78dfe1c1bc11da131a4d9ade124a71dc6e268049539a47837e51c3b92b91e
Sep 10 15:54:34.398947 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:34.398843    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v" event={"ID":"da45bba5-1d5b-4a1c-8ff2-61bead26ab19","Type":"ContainerStarted","Data":"0609285d9012e47d49445f76f6e3ffa5444f9fd3cbe7fceb1493e8aa7ee3b877"}
Sep 10 15:54:34.398947 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:34.398919    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v" event={"ID":"da45bba5-1d5b-4a1c-8ff2-61bead26ab19","Type":"ContainerStarted","Data":"13a78dfe1c1bc11da131a4d9ade124a71dc6e268049539a47837e51c3b92b91e"}
Sep 10 15:54:34.398947 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:34.398947    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v"
Sep 10 15:54:34.419113 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:34.419062    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v" podStartSLOduration=1.419044746 podStartE2EDuration="1.419044746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:34.417979777 +0000 UTC m=+753.222299040" watchObservedRunningTime="2026-09-10 15:54:34.419044746 +0000 UTC m=+753.223364050"
Sep 10 15:54:36.346310 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.346276    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"]
Sep 10 15:54:36.350703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.350683    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.353043 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.353020    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-87a024-openshift-default-dockercfg-wv4bt\""
Sep 10 15:54:36.363619 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.363591    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"]
Sep 10 15:54:36.510314 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.510279    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-podinfo\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.510314 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.510321    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-envoy\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.510567 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.510347    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-token\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.510567 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.510443    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-data\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.510567 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.510481    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-credential-socket\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.510567 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.510535    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-workload-certs\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.510779 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.510642    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8qdt\" (UniqueName: \"kubernetes.io/projected/e5116818-0a78-4fe9-8b84-b587b1a3d080-kube-api-access-t8qdt\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.510779 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.510684    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5116818-0a78-4fe9-8b84-b587b1a3d080-istiod-ca-cert\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.510779 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.510709    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-workload-socket\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.612275 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.612178    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-data\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.612275 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.612225    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-credential-socket\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.612275 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.612249    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-workload-certs\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.612569 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.612294    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8qdt\" (UniqueName: \"kubernetes.io/projected/e5116818-0a78-4fe9-8b84-b587b1a3d080-kube-api-access-t8qdt\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.612569 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.612327    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5116818-0a78-4fe9-8b84-b587b1a3d080-istiod-ca-cert\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.612569 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.612350    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-workload-socket\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.612569 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.612427    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-podinfo\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.612569 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.612459    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-envoy\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.612569 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.612485    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-token\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.612843 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.612610    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-data\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.612843 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.612625    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-workload-certs\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.612843 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.612743    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-credential-socket\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.612942 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.612849    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-workload-socket\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.613104 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.613081    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5116818-0a78-4fe9-8b84-b587b1a3d080-istiod-ca-cert\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.615284 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.615260    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-podinfo\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.615417 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.615346    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-envoy\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.620780 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.620757    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-token\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.621236 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.621214    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8qdt\" (UniqueName: \"kubernetes.io/projected/e5116818-0a78-4fe9-8b84-b587b1a3d080-kube-api-access-t8qdt\") pod \"e2e-autores-87a024-openshift-default-8554f8f878-j4fjs\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") " pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.664147 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.664107    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:36.948880 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.948856    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"]
Sep 10 15:54:36.950826 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:54:36.950794    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5116818_0a78_4fe9_8b84_b587b1a3d080.slice/crio-97aeec8a290bf4e999e2fe29b1a2a415a1451d7d134e037aae18b658127f8a5f WatchSource:0}: Error finding container 97aeec8a290bf4e999e2fe29b1a2a415a1451d7d134e037aae18b658127f8a5f: Status 404 returned error can't find the container with id 97aeec8a290bf4e999e2fe29b1a2a415a1451d7d134e037aae18b658127f8a5f
Sep 10 15:54:36.953229 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.953196    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:54:36.953309 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.953268    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:54:36.953309 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:36.953295    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:54:37.418430 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:37.418398    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs" event={"ID":"e5116818-0a78-4fe9-8b84-b587b1a3d080","Type":"ContainerStarted","Data":"a2a768687934982ce613cb10c742937c23a62035408ab610c1acfa3bf9152d37"}
Sep 10 15:54:37.418855 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:37.418457    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs" event={"ID":"e5116818-0a78-4fe9-8b84-b587b1a3d080","Type":"ContainerStarted","Data":"97aeec8a290bf4e999e2fe29b1a2a415a1451d7d134e037aae18b658127f8a5f"}
Sep 10 15:54:37.445757 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:37.445696    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs" podStartSLOduration=1.445679366 podStartE2EDuration="1.445679366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:37.444239778 +0000 UTC m=+756.248559039" watchObservedRunningTime="2026-09-10 15:54:37.445679366 +0000 UTC m=+756.249998631"
Sep 10 15:54:37.665090 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:37.665052    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:38.670012 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:38.669982    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:39.433722 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:39.433683    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:39.434782 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:39.434756    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:40.860707 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:40.860676    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq"]
Sep 10 15:54:40.864322 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:40.864305    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq"
Sep 10 15:54:40.866793 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:40.866772    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-54da81e9-dockercfg-wmtn4\""
Sep 10 15:54:40.866996 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:40.866861    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-54da81e9\""
Sep 10 15:54:40.874190 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:40.873975    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq"]
Sep 10 15:54:40.959726 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:40.959690    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwn9r\" (UniqueName: \"kubernetes.io/projected/e5170c13-5668-4174-bcc2-13801753cff8-kube-api-access-mwn9r\") pod \"payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq\" (UID: \"e5170c13-5668-4174-bcc2-13801753cff8\") " pod="openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq"
Sep 10 15:54:40.959968 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:40.959954    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5170c13-5668-4174-bcc2-13801753cff8-plugins-config-volume\") pod \"payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq\" (UID: \"e5170c13-5668-4174-bcc2-13801753cff8\") " pod="openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq"
Sep 10 15:54:41.060713 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:41.060675    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwn9r\" (UniqueName: \"kubernetes.io/projected/e5170c13-5668-4174-bcc2-13801753cff8-kube-api-access-mwn9r\") pod \"payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq\" (UID: \"e5170c13-5668-4174-bcc2-13801753cff8\") " pod="openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq"
Sep 10 15:54:41.060921 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:41.060738    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5170c13-5668-4174-bcc2-13801753cff8-plugins-config-volume\") pod \"payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq\" (UID: \"e5170c13-5668-4174-bcc2-13801753cff8\") " pod="openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq"
Sep 10 15:54:41.061698 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:41.061672    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5170c13-5668-4174-bcc2-13801753cff8-plugins-config-volume\") pod \"payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq\" (UID: \"e5170c13-5668-4174-bcc2-13801753cff8\") " pod="openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq"
Sep 10 15:54:41.070832 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:41.070803    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwn9r\" (UniqueName: \"kubernetes.io/projected/e5170c13-5668-4174-bcc2-13801753cff8-kube-api-access-mwn9r\") pod \"payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq\" (UID: \"e5170c13-5668-4174-bcc2-13801753cff8\") " pod="openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq"
Sep 10 15:54:41.177270 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:41.177244    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq"
Sep 10 15:54:41.483353 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:41.483080    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq"]
Sep 10 15:54:41.485485 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:54:41.485461    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5170c13_5668_4174_bcc2_13801753cff8.slice/crio-bf9fba5884b87dbe823dc82350fcbf6cd461c3dcf1b83e32fd36e03c85015575 WatchSource:0}: Error finding container bf9fba5884b87dbe823dc82350fcbf6cd461c3dcf1b83e32fd36e03c85015575: Status 404 returned error can't find the container with id bf9fba5884b87dbe823dc82350fcbf6cd461c3dcf1b83e32fd36e03c85015575
Sep 10 15:54:41.876968 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:41.876885    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm"]
Sep 10 15:54:41.880756 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:41.880736    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm"
Sep 10 15:54:41.883144 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:41.883122    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-87a024\""
Sep 10 15:54:41.893006 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:41.892918    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm"]
Sep 10 15:54:42.070487 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:42.070456    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqhhd\" (UniqueName: \"kubernetes.io/projected/0fc405ea-b5f4-4c99-ba4c-bc98aac499ea-kube-api-access-hqhhd\") pod \"payload-processing-e2e-autores-87a024-6bd4c565-bwglm\" (UID: \"0fc405ea-b5f4-4c99-ba4c-bc98aac499ea\") " pod="openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm"
Sep 10 15:54:42.070685 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:42.070518    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc405ea-b5f4-4c99-ba4c-bc98aac499ea-plugins-config-volume\") pod \"payload-processing-e2e-autores-87a024-6bd4c565-bwglm\" (UID: \"0fc405ea-b5f4-4c99-ba4c-bc98aac499ea\") " pod="openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm"
Sep 10 15:54:42.171900 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:42.171865    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc405ea-b5f4-4c99-ba4c-bc98aac499ea-plugins-config-volume\") pod \"payload-processing-e2e-autores-87a024-6bd4c565-bwglm\" (UID: \"0fc405ea-b5f4-4c99-ba4c-bc98aac499ea\") " pod="openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm"
Sep 10 15:54:42.172082 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:42.171988    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqhhd\" (UniqueName: \"kubernetes.io/projected/0fc405ea-b5f4-4c99-ba4c-bc98aac499ea-kube-api-access-hqhhd\") pod \"payload-processing-e2e-autores-87a024-6bd4c565-bwglm\" (UID: \"0fc405ea-b5f4-4c99-ba4c-bc98aac499ea\") " pod="openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm"
Sep 10 15:54:42.172646 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:42.172623    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc405ea-b5f4-4c99-ba4c-bc98aac499ea-plugins-config-volume\") pod \"payload-processing-e2e-autores-87a024-6bd4c565-bwglm\" (UID: \"0fc405ea-b5f4-4c99-ba4c-bc98aac499ea\") " pod="openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm"
Sep 10 15:54:42.180339 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:42.180311    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqhhd\" (UniqueName: \"kubernetes.io/projected/0fc405ea-b5f4-4c99-ba4c-bc98aac499ea-kube-api-access-hqhhd\") pod \"payload-processing-e2e-autores-87a024-6bd4c565-bwglm\" (UID: \"0fc405ea-b5f4-4c99-ba4c-bc98aac499ea\") " pod="openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm"
Sep 10 15:54:42.196044 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:42.196019    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm"
Sep 10 15:54:42.453075 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:42.452998    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq" event={"ID":"e5170c13-5668-4174-bcc2-13801753cff8","Type":"ContainerStarted","Data":"ef89bf3811773685ed988be758a69afe3659127bf2e8947b85976a7f06409439"}
Sep 10 15:54:42.453075 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:42.453047    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq" event={"ID":"e5170c13-5668-4174-bcc2-13801753cff8","Type":"ContainerStarted","Data":"bf9fba5884b87dbe823dc82350fcbf6cd461c3dcf1b83e32fd36e03c85015575"}
Sep 10 15:54:42.453286 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:42.453079    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq"
Sep 10 15:54:42.471009 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:42.470950    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq" podStartSLOduration=2.470930814 podStartE2EDuration="2.470930814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:42.469820527 +0000 UTC m=+761.274139799" watchObservedRunningTime="2026-09-10 15:54:42.470930814 +0000 UTC m=+761.275250077"
Sep 10 15:54:42.510199 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:42.510140    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm"]
Sep 10 15:54:42.520663 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:54:42.520626    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc405ea_b5f4_4c99_ba4c_bc98aac499ea.slice/crio-3f36ada16da06678a9f66515e665a913c4e4abaf1f812100bf20e266275c563d WatchSource:0}: Error finding container 3f36ada16da06678a9f66515e665a913c4e4abaf1f812100bf20e266275c563d: Status 404 returned error can't find the container with id 3f36ada16da06678a9f66515e665a913c4e4abaf1f812100bf20e266275c563d
Sep 10 15:54:43.461020 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:43.460982    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm" event={"ID":"0fc405ea-b5f4-4c99-ba4c-bc98aac499ea","Type":"ContainerStarted","Data":"4fd0abac3fbb4c3c7730a3bd088979a1642e1579b6dea8cc6f21678fe280c762"}
Sep 10 15:54:43.461514 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:43.461189    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm"
Sep 10 15:54:43.461514 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:43.461227    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm" event={"ID":"0fc405ea-b5f4-4c99-ba4c-bc98aac499ea","Type":"ContainerStarted","Data":"3f36ada16da06678a9f66515e665a913c4e4abaf1f812100bf20e266275c563d"}
Sep 10 15:54:43.479651 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:43.479601    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm" podStartSLOduration=2.479584384 podStartE2EDuration="2.479584384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:43.476968275 +0000 UTC m=+762.281287537" watchObservedRunningTime="2026-09-10 15:54:43.479584384 +0000 UTC m=+762.283903645"
Sep 10 15:54:45.406980 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:45.406944    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v"
Sep 10 15:54:49.499085 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:49.499042    2564 generic.go:356] "Generic (PLEG): container finished" podID="3765fb24-410e-4107-856c-af41b073836b" containerID="f04919c4dfd1a5a3ad150812cc7b6f28cc1f16a01dbbb5ae076410c8f63527c6" exitCode=0
Sep 10 15:54:49.499645 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:49.499220    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg" event={"ID":"3765fb24-410e-4107-856c-af41b073836b","Type":"ContainerDied","Data":"f04919c4dfd1a5a3ad150812cc7b6f28cc1f16a01dbbb5ae076410c8f63527c6"}
Sep 10 15:54:49.499718 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:49.499696    2564 scope.go:117] "RemoveContainer" containerID="f04919c4dfd1a5a3ad150812cc7b6f28cc1f16a01dbbb5ae076410c8f63527c6"
Sep 10 15:54:49.501479 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:49.501438    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:54:49.501730 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:49.501701    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:54:49.501897 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:49.501876    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:54:50.510325 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:50.510277    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg" event={"ID":"3765fb24-410e-4107-856c-af41b073836b","Type":"ContainerStarted","Data":"2318776fc92194de041ed8b497306e3161c6da995615200c94fce80d7b08d5d0"}
Sep 10 15:54:51.020228 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:51.020195    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:51.020606 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:51.020513    2564 patch_prober.go:28] interesting pod/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 10 15:54:51.020768 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:51.020745    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg" podUID="3765fb24-410e-4107-856c-af41b073836b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 10 15:54:52.020702 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:52.020656    2564 patch_prober.go:28] interesting pod/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 10 15:54:52.021226 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:52.020744    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg" podUID="3765fb24-410e-4107-856c-af41b073836b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 10 15:54:53.020659 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:53.020618    2564 patch_prober.go:28] interesting pod/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 10 15:54:53.020871 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:53.020691    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg" podUID="3765fb24-410e-4107-856c-af41b073836b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 10 15:54:53.463933 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:53.463905    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq"
Sep 10 15:54:54.020698 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:54.020650    2564 patch_prober.go:28] interesting pod/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 10 15:54:54.020902 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:54.020736    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg" podUID="3765fb24-410e-4107-856c-af41b073836b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 10 15:54:54.468950 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:54.468919    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm"
Sep 10 15:54:55.020114 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:55.020066    2564 patch_prober.go:28] interesting pod/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 10 15:54:55.020342 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:55.020142    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg" podUID="3765fb24-410e-4107-856c-af41b073836b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 10 15:54:55.475328 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:55.475288    2564 patch_prober.go:28] interesting pod/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 15:54:55.475798 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:55.475350    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs" podUID="e5116818-0a78-4fe9-8b84-b587b1a3d080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 15:54:56.020498 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:56.020461    2564 patch_prober.go:28] interesting pod/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 10 15:54:56.020708 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:56.020518    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg" podUID="3765fb24-410e-4107-856c-af41b073836b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 10 15:54:57.024673 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:57.024640    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:57.025041 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:57.024939    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:57.025630 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:57.025613    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 15:54:58.576755 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:58.576723    2564 generic.go:356] "Generic (PLEG): container finished" podID="e5116818-0a78-4fe9-8b84-b587b1a3d080" containerID="a2a768687934982ce613cb10c742937c23a62035408ab610c1acfa3bf9152d37" exitCode=0
Sep 10 15:54:58.577216 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:58.576802    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs" event={"ID":"e5116818-0a78-4fe9-8b84-b587b1a3d080","Type":"ContainerDied","Data":"a2a768687934982ce613cb10c742937c23a62035408ab610c1acfa3bf9152d37"}
Sep 10 15:54:58.577403 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:58.577382    2564 scope.go:117] "RemoveContainer" containerID="a2a768687934982ce613cb10c742937c23a62035408ab610c1acfa3bf9152d37"
Sep 10 15:54:58.578528 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:58.578501    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:54:58.578614 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:58.578581    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:54:58.578653 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:58.578613    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:54:59.584103 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:59.584074    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs" event={"ID":"e5116818-0a78-4fe9-8b84-b587b1a3d080","Type":"ContainerStarted","Data":"330ae4ba68b3a6f8c2ce7432a42189f9e63116fa6c42c611f65044fd65a506ff"}
Sep 10 15:54:59.664954 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:59.664922    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:54:59.665137 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:59.665115    2564 patch_prober.go:28] interesting pod/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 10 15:54:59.665222 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:54:59.665196    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs" podUID="e5116818-0a78-4fe9-8b84-b587b1a3d080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 10 15:55:00.665498 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:00.665455    2564 patch_prober.go:28] interesting pod/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 10 15:55:00.665973 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:00.665527    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs" podUID="e5116818-0a78-4fe9-8b84-b587b1a3d080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 10 15:55:01.665407 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:01.665367    2564 patch_prober.go:28] interesting pod/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 10 15:55:01.665618 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:01.665427    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs" podUID="e5116818-0a78-4fe9-8b84-b587b1a3d080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 10 15:55:02.665091 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:02.665010    2564 patch_prober.go:28] interesting pod/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 10 15:55:02.665091 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:02.665066    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs" podUID="e5116818-0a78-4fe9-8b84-b587b1a3d080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 10 15:55:03.665461 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:03.665420    2564 patch_prober.go:28] interesting pod/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 10 15:55:03.665931 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:03.665489    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs" podUID="e5116818-0a78-4fe9-8b84-b587b1a3d080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 10 15:55:04.231736 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:04.231705    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54da81e9-96jv5"]
Sep 10 15:55:04.235661 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:04.235643    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54da81e9-96jv5"
Sep 10 15:55:04.238001 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:04.237980    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2wghg\""
Sep 10 15:55:04.243360 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:04.242887    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54da81e9-96jv5"]
Sep 10 15:55:04.332236 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:04.332198    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32573ae-792f-4a4e-b518-746c88857828-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-54da81e9-96jv5\" (UID: \"b32573ae-792f-4a4e-b518-746c88857828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54da81e9-96jv5"
Sep 10 15:55:04.433289 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:04.433255    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32573ae-792f-4a4e-b518-746c88857828-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-54da81e9-96jv5\" (UID: \"b32573ae-792f-4a4e-b518-746c88857828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54da81e9-96jv5"
Sep 10 15:55:04.433901 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:04.433879    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32573ae-792f-4a4e-b518-746c88857828-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-54da81e9-96jv5\" (UID: \"b32573ae-792f-4a4e-b518-746c88857828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54da81e9-96jv5"
Sep 10 15:55:04.548674 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:04.548573    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54da81e9-96jv5"
Sep 10 15:55:04.664588 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:04.664556    2564 patch_prober.go:28] interesting pod/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 10 15:55:04.664728 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:04.664621    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs" podUID="e5116818-0a78-4fe9-8b84-b587b1a3d080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 10 15:55:04.841946 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:04.841919    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54da81e9-96jv5"]
Sep 10 15:55:04.844859 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:55:04.844830    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32573ae_792f_4a4e_b518_746c88857828.slice/crio-8aea5154bb911a2324dcc3fece0e5af8be0ec7fd8c00f37c120afa2108e9ff6b WatchSource:0}: Error finding container 8aea5154bb911a2324dcc3fece0e5af8be0ec7fd8c00f37c120afa2108e9ff6b: Status 404 returned error can't find the container with id 8aea5154bb911a2324dcc3fece0e5af8be0ec7fd8c00f37c120afa2108e9ff6b
Sep 10 15:55:05.661141 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:05.661101    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54da81e9-96jv5" event={"ID":"b32573ae-792f-4a4e-b518-746c88857828","Type":"ContainerStarted","Data":"8aea5154bb911a2324dcc3fece0e5af8be0ec7fd8c00f37c120afa2108e9ff6b"}
Sep 10 15:55:05.672591 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:05.672561    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:55:05.672938 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:05.672919    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:55:05.673632 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:05.673611    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:55:07.675815 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:07.675782    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54da81e9-96jv5" event={"ID":"b32573ae-792f-4a4e-b518-746c88857828","Type":"ContainerStarted","Data":"ccb57d457d1d1df631b7fd477fca05a44c89204b20d3c95a11d211f255f0baef"}
Sep 10 15:55:07.695291 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:07.695231    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54da81e9-96jv5" podStartSLOduration=1.232752115 podStartE2EDuration="3.695212768s" totalImagesPullingTime="2.462460653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:07.694269513 +0000 UTC m=+786.498588776" watchObservedRunningTime="2026-09-10 15:55:07.695212768 +0000 UTC m=+786.499532030"
Sep 10 15:55:08.683105 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:08.683068    2564 generic.go:356] "Generic (PLEG): container finished" podID="b32573ae-792f-4a4e-b518-746c88857828" containerID="ccb57d457d1d1df631b7fd477fca05a44c89204b20d3c95a11d211f255f0baef" exitCode=0
Sep 10 15:55:08.683536 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:08.683179    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54da81e9-96jv5" event={"ID":"b32573ae-792f-4a4e-b518-746c88857828","Type":"ContainerDied","Data":"ccb57d457d1d1df631b7fd477fca05a44c89204b20d3c95a11d211f255f0baef"}
Sep 10 15:55:09.836678 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:09.836653    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54da81e9-96jv5"
Sep 10 15:55:09.995954 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:09.995865    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32573ae-792f-4a4e-b518-746c88857828-maas-api-service-ca\") pod \"b32573ae-792f-4a4e-b518-746c88857828\" (UID: \"b32573ae-792f-4a4e-b518-746c88857828\") "
Sep 10 15:55:09.996313 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:09.996273    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b32573ae-792f-4a4e-b518-746c88857828-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b32573ae-792f-4a4e-b518-746c88857828" (UID: "b32573ae-792f-4a4e-b518-746c88857828"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:55:10.097152 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:10.097111    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32573ae-792f-4a4e-b518-746c88857828-maas-api-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:55:10.695130 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:10.695096    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54da81e9-96jv5"
Sep 10 15:55:10.695130 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:10.695107    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54da81e9-96jv5" event={"ID":"b32573ae-792f-4a4e-b518-746c88857828","Type":"ContainerDied","Data":"8aea5154bb911a2324dcc3fece0e5af8be0ec7fd8c00f37c120afa2108e9ff6b"}
Sep 10 15:55:10.695399 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:10.695177    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8aea5154bb911a2324dcc3fece0e5af8be0ec7fd8c00f37c120afa2108e9ff6b"
Sep 10 15:55:21.087580 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:21.087531    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq"]
Sep 10 15:55:21.088214 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:21.087836    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq" podUID="e5170c13-5668-4174-bcc2-13801753cff8" containerName="payload-processing" containerID="cri-o://ef89bf3811773685ed988be758a69afe3659127bf2e8947b85976a7f06409439" gracePeriod=30
Sep 10 15:55:21.286475 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:21.286378    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54da81e9-96jv5"]
Sep 10 15:55:21.290363 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:21.290313    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-54da81e9-96jv5"]
Sep 10 15:55:21.360981 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:21.360956    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq"
Sep 10 15:55:21.419439 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:21.419407    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5170c13-5668-4174-bcc2-13801753cff8-plugins-config-volume\") pod \"e5170c13-5668-4174-bcc2-13801753cff8\" (UID: \"e5170c13-5668-4174-bcc2-13801753cff8\") "
Sep 10 15:55:21.419628 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:21.419610    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwn9r\" (UniqueName: \"kubernetes.io/projected/e5170c13-5668-4174-bcc2-13801753cff8-kube-api-access-mwn9r\") pod \"e5170c13-5668-4174-bcc2-13801753cff8\" (UID: \"e5170c13-5668-4174-bcc2-13801753cff8\") "
Sep 10 15:55:21.419894 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:21.419861    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5170c13-5668-4174-bcc2-13801753cff8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e5170c13-5668-4174-bcc2-13801753cff8" (UID: "e5170c13-5668-4174-bcc2-13801753cff8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:55:21.420102 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:21.420051    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5170c13-5668-4174-bcc2-13801753cff8-plugins-config-volume\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:55:21.423121 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:21.423050    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5170c13-5668-4174-bcc2-13801753cff8-kube-api-access-mwn9r" (OuterVolumeSpecName: "kube-api-access-mwn9r") pod "e5170c13-5668-4174-bcc2-13801753cff8" (UID: "e5170c13-5668-4174-bcc2-13801753cff8"). InnerVolumeSpecName "kube-api-access-mwn9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:55:21.521534 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:21.521503    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwn9r\" (UniqueName: \"kubernetes.io/projected/e5170c13-5668-4174-bcc2-13801753cff8-kube-api-access-mwn9r\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:55:21.673618 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:21.673585    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b32573ae-792f-4a4e-b518-746c88857828" path="/var/lib/kubelet/pods/b32573ae-792f-4a4e-b518-746c88857828/volumes"
Sep 10 15:55:21.756572 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:21.756536    2564 generic.go:356] "Generic (PLEG): container finished" podID="e5170c13-5668-4174-bcc2-13801753cff8" containerID="ef89bf3811773685ed988be758a69afe3659127bf2e8947b85976a7f06409439" exitCode=0
Sep 10 15:55:21.756572 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:21.756579    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq"
Sep 10 15:55:21.756829 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:21.756636    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq" event={"ID":"e5170c13-5668-4174-bcc2-13801753cff8","Type":"ContainerDied","Data":"ef89bf3811773685ed988be758a69afe3659127bf2e8947b85976a7f06409439"}
Sep 10 15:55:21.756829 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:21.756684    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq" event={"ID":"e5170c13-5668-4174-bcc2-13801753cff8","Type":"ContainerDied","Data":"bf9fba5884b87dbe823dc82350fcbf6cd461c3dcf1b83e32fd36e03c85015575"}
Sep 10 15:55:21.756829 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:21.756705    2564 scope.go:117] "RemoveContainer" containerID="ef89bf3811773685ed988be758a69afe3659127bf2e8947b85976a7f06409439"
Sep 10 15:55:21.765810 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:21.765792    2564 scope.go:117] "RemoveContainer" containerID="ef89bf3811773685ed988be758a69afe3659127bf2e8947b85976a7f06409439"
Sep 10 15:55:21.766081 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:55:21.766059    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef89bf3811773685ed988be758a69afe3659127bf2e8947b85976a7f06409439\": container with ID starting with ef89bf3811773685ed988be758a69afe3659127bf2e8947b85976a7f06409439 not found: ID does not exist" containerID="ef89bf3811773685ed988be758a69afe3659127bf2e8947b85976a7f06409439"
Sep 10 15:55:21.766177 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:21.766092    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef89bf3811773685ed988be758a69afe3659127bf2e8947b85976a7f06409439"} err="failed to get container status \"ef89bf3811773685ed988be758a69afe3659127bf2e8947b85976a7f06409439\": rpc error: code = NotFound desc = could not find container \"ef89bf3811773685ed988be758a69afe3659127bf2e8947b85976a7f06409439\": container with ID starting with ef89bf3811773685ed988be758a69afe3659127bf2e8947b85976a7f06409439 not found: ID does not exist"
Sep 10 15:55:21.783696 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:21.783658    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq"]
Sep 10 15:55:21.788152 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:21.788121    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-54da81e9-5678cf7fdf-qdxrq"]
Sep 10 15:55:22.403606 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.403574    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"]
Sep 10 15:55:22.404125 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.404108    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5170c13-5668-4174-bcc2-13801753cff8" containerName="payload-processing"
Sep 10 15:55:22.404125 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.404127    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5170c13-5668-4174-bcc2-13801753cff8" containerName="payload-processing"
Sep 10 15:55:22.404251 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.404138    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b32573ae-792f-4a4e-b518-746c88857828" containerName="revoke-keys"
Sep 10 15:55:22.404251 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.404144    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b32573ae-792f-4a4e-b518-746c88857828" containerName="revoke-keys"
Sep 10 15:55:22.404324 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.404263    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5170c13-5668-4174-bcc2-13801753cff8" containerName="payload-processing"
Sep 10 15:55:22.404324 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.404274    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b32573ae-792f-4a4e-b518-746c88857828" containerName="revoke-keys"
Sep 10 15:55:22.406987 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.406940    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.409839 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.409820    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-384e8433-gw-openshift-default-dockercfg-f4wws\""
Sep 10 15:55:22.419922 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.419897    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"]
Sep 10 15:55:22.531702 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.531625    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-workload-socket\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.531888 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.531701    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-data\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.531888 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.531815    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-token\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.532029 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.531887    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfjwq\" (UniqueName: \"kubernetes.io/projected/fb08e393-d2da-47de-bab0-c7019b7e4ec8-kube-api-access-sfjwq\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.532029 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.531917    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-credential-socket\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.532029 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.531983    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-workload-certs\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.532029 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.532005    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-envoy\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.532029 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.532029    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-podinfo\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.532284 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.532128    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istiod-ca-cert\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.632831 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.632790    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istiod-ca-cert\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.632831 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.632834    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-workload-socket\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.633090 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.632872    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-data\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.633090 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.632901    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-token\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.633090 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.632931    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfjwq\" (UniqueName: \"kubernetes.io/projected/fb08e393-d2da-47de-bab0-c7019b7e4ec8-kube-api-access-sfjwq\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.633090 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.632949    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-credential-socket\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.633090 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.632984    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-workload-certs\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.633090 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.633005    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-envoy\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.633090 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.633028    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-podinfo\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.633497 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.633441    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-credential-socket\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.633630 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.633602    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-workload-socket\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.633713 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.633638    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-workload-certs\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.633713 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.633648    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-data\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.633713 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.633687    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istiod-ca-cert\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.636196 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.636135    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-envoy\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.636298 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.636270    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-podinfo\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.644081 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.644052    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-token\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.645667 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.645641    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfjwq\" (UniqueName: \"kubernetes.io/projected/fb08e393-d2da-47de-bab0-c7019b7e4ec8-kube-api-access-sfjwq\") pod \"e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") " pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:22.719632 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:22.719583    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:23.005311 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:23.005285    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"]
Sep 10 15:55:23.007734 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:55:23.007706    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb08e393_d2da_47de_bab0_c7019b7e4ec8.slice/crio-edc590fa199f6dd5328300947efe614cacfc72d093c2636fde16ce5360621ec7 WatchSource:0}: Error finding container edc590fa199f6dd5328300947efe614cacfc72d093c2636fde16ce5360621ec7: Status 404 returned error can't find the container with id edc590fa199f6dd5328300947efe614cacfc72d093c2636fde16ce5360621ec7
Sep 10 15:55:23.010955 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:23.010894    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:55:23.011086 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:23.010995    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:55:23.011086 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:23.011039    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:55:23.675784 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:23.675747    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5170c13-5668-4174-bcc2-13801753cff8" path="/var/lib/kubelet/pods/e5170c13-5668-4174-bcc2-13801753cff8/volumes"
Sep 10 15:55:23.773756 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:23.773719    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh" event={"ID":"fb08e393-d2da-47de-bab0-c7019b7e4ec8","Type":"ContainerStarted","Data":"d4a4c9687c0ad7b283610f0ff0b3cec2129f56c2cb08123baec6719e8ebfcb50"}
Sep 10 15:55:23.773917 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:23.773770    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh" event={"ID":"fb08e393-d2da-47de-bab0-c7019b7e4ec8","Type":"ContainerStarted","Data":"edc590fa199f6dd5328300947efe614cacfc72d093c2636fde16ce5360621ec7"}
Sep 10 15:55:23.802949 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:23.802888    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh" podStartSLOduration=1.8028646350000002 podStartE2EDuration="1.802864635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:23.801372364 +0000 UTC m=+802.605691635" watchObservedRunningTime="2026-09-10 15:55:23.802864635 +0000 UTC m=+802.607183899"
Sep 10 15:55:24.720721 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:24.720685    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:24.726242 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:24.726216    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:24.780024 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:24.779995    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:24.781117 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:24.781099    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:46.914934 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:46.914898    2564 generic.go:356] "Generic (PLEG): container finished" podID="fb08e393-d2da-47de-bab0-c7019b7e4ec8" containerID="d4a4c9687c0ad7b283610f0ff0b3cec2129f56c2cb08123baec6719e8ebfcb50" exitCode=0
Sep 10 15:55:46.915381 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:46.914972    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh" event={"ID":"fb08e393-d2da-47de-bab0-c7019b7e4ec8","Type":"ContainerDied","Data":"d4a4c9687c0ad7b283610f0ff0b3cec2129f56c2cb08123baec6719e8ebfcb50"}
Sep 10 15:55:46.915426 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:46.915401    2564 scope.go:117] "RemoveContainer" containerID="d4a4c9687c0ad7b283610f0ff0b3cec2129f56c2cb08123baec6719e8ebfcb50"
Sep 10 15:55:46.916631 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:46.916599    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:55:46.916712 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:46.916678    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:55:46.916750 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:46.916711    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:55:47.928089 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:47.928055    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh" event={"ID":"fb08e393-d2da-47de-bab0-c7019b7e4ec8","Type":"ContainerStarted","Data":"3a481b0079e8e2c04eba281829dd959094b6084d80b8b11a98f6aabbeb7f868c"}
Sep 10 15:55:48.719984 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:48.719951    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:48.720358 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:48.720323    2564 patch_prober.go:28] interesting pod/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 10 15:55:48.720537 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:48.720513    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh" podUID="fb08e393-d2da-47de-bab0-c7019b7e4ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 10 15:55:49.720806 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:49.720765    2564 patch_prober.go:28] interesting pod/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 10 15:55:49.721480 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:49.721406    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh" podUID="fb08e393-d2da-47de-bab0-c7019b7e4ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 10 15:55:50.279628 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:50.279591    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-384e8433-95hgb"]
Sep 10 15:55:50.283292 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:50.283276    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-384e8433-95hgb"
Sep 10 15:55:50.286092 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:50.286072    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2wghg\""
Sep 10 15:55:50.293636 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:50.293611    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-384e8433-95hgb"]
Sep 10 15:55:50.340058 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:50.340006    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/654d6d0e-a57b-49ef-b48c-b98c66559655-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-384e8433-95hgb\" (UID: \"654d6d0e-a57b-49ef-b48c-b98c66559655\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-384e8433-95hgb"
Sep 10 15:55:50.441339 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:50.441303    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/654d6d0e-a57b-49ef-b48c-b98c66559655-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-384e8433-95hgb\" (UID: \"654d6d0e-a57b-49ef-b48c-b98c66559655\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-384e8433-95hgb"
Sep 10 15:55:50.441937 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:50.441916    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/654d6d0e-a57b-49ef-b48c-b98c66559655-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-384e8433-95hgb\" (UID: \"654d6d0e-a57b-49ef-b48c-b98c66559655\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-384e8433-95hgb"
Sep 10 15:55:50.595134 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:50.595036    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-384e8433-95hgb"
Sep 10 15:55:50.723725 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:50.723656    2564 patch_prober.go:28] interesting pod/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 10 15:55:50.724241 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:50.723735    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh" podUID="fb08e393-d2da-47de-bab0-c7019b7e4ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 10 15:55:50.909107 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:50.908994    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-384e8433-95hgb"]
Sep 10 15:55:50.921826 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:55:50.921773    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654d6d0e_a57b_49ef_b48c_b98c66559655.slice/crio-596934a85acc0daa76633bcd69dc94565058c8783a8d76600824df0fb049abfb WatchSource:0}: Error finding container 596934a85acc0daa76633bcd69dc94565058c8783a8d76600824df0fb049abfb: Status 404 returned error can't find the container with id 596934a85acc0daa76633bcd69dc94565058c8783a8d76600824df0fb049abfb
Sep 10 15:55:50.954123 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:50.954096    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-384e8433-95hgb" event={"ID":"654d6d0e-a57b-49ef-b48c-b98c66559655","Type":"ContainerStarted","Data":"596934a85acc0daa76633bcd69dc94565058c8783a8d76600824df0fb049abfb"}
Sep 10 15:55:51.720382 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:51.720343    2564 patch_prober.go:28] interesting pod/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 10 15:55:51.720677 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:51.720643    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh" podUID="fb08e393-d2da-47de-bab0-c7019b7e4ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 10 15:55:51.961620 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:51.961582    2564 generic.go:356] "Generic (PLEG): container finished" podID="654d6d0e-a57b-49ef-b48c-b98c66559655" containerID="9dccc38a2a5207580cc551ea9260395e364f3423e05866618fe2b4cf5165ba8f" exitCode=0
Sep 10 15:55:51.962045 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:51.961668    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-384e8433-95hgb" event={"ID":"654d6d0e-a57b-49ef-b48c-b98c66559655","Type":"ContainerDied","Data":"9dccc38a2a5207580cc551ea9260395e364f3423e05866618fe2b4cf5165ba8f"}
Sep 10 15:55:52.720185 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:52.720117    2564 patch_prober.go:28] interesting pod/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 10 15:55:52.720388 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:52.720205    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh" podUID="fb08e393-d2da-47de-bab0-c7019b7e4ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 10 15:55:53.157711 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:53.157684    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-384e8433-95hgb"
Sep 10 15:55:53.272493 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:53.272456    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/654d6d0e-a57b-49ef-b48c-b98c66559655-maas-api-service-ca\") pod \"654d6d0e-a57b-49ef-b48c-b98c66559655\" (UID: \"654d6d0e-a57b-49ef-b48c-b98c66559655\") "
Sep 10 15:55:53.272820 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:53.272791    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/654d6d0e-a57b-49ef-b48c-b98c66559655-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "654d6d0e-a57b-49ef-b48c-b98c66559655" (UID: "654d6d0e-a57b-49ef-b48c-b98c66559655"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:55:53.373908 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:53.373815    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/654d6d0e-a57b-49ef-b48c-b98c66559655-maas-api-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:55:53.725345 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:53.725312    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:53.725723 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:53.725705    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:53.726366 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:53.726340    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:55:53.974515 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:53.974440    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-384e8433-95hgb"
Sep 10 15:55:53.974515 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:53.974472    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-384e8433-95hgb" event={"ID":"654d6d0e-a57b-49ef-b48c-b98c66559655","Type":"ContainerDied","Data":"596934a85acc0daa76633bcd69dc94565058c8783a8d76600824df0fb049abfb"}
Sep 10 15:55:53.974695 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:53.974524    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="596934a85acc0daa76633bcd69dc94565058c8783a8d76600824df0fb049abfb"
Sep 10 15:55:57.408261 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:57.408217    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-384e8433-95hgb"]
Sep 10 15:55:57.410648 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:57.410626    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-384e8433-95hgb"]
Sep 10 15:55:57.542453 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:57.542423    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"]
Sep 10 15:55:57.542658 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:57.542631    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh" podUID="fb08e393-d2da-47de-bab0-c7019b7e4ec8" containerName="istio-proxy" containerID="cri-o://3a481b0079e8e2c04eba281829dd959094b6084d80b8b11a98f6aabbeb7f868c" gracePeriod=30
Sep 10 15:55:57.672770 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:55:57.672740    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="654d6d0e-a57b-49ef-b48c-b98c66559655" path="/var/lib/kubelet/pods/654d6d0e-a57b-49ef-b48c-b98c66559655/volumes"
Sep 10 15:56:02.802721 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.802693    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:56:02.878109 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.878081    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-data\") pod \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") "
Sep 10 15:56:02.878305 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.878123    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-credential-socket\") pod \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") "
Sep 10 15:56:02.878305 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.878187    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-podinfo\") pod \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") "
Sep 10 15:56:02.878305 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.878212    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfjwq\" (UniqueName: \"kubernetes.io/projected/fb08e393-d2da-47de-bab0-c7019b7e4ec8-kube-api-access-sfjwq\") pod \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") "
Sep 10 15:56:02.878305 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.878238    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-envoy\") pod \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") "
Sep 10 15:56:02.878551 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.878344    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-workload-socket\") pod \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") "
Sep 10 15:56:02.878551 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.878386    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-token\") pod \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") "
Sep 10 15:56:02.878551 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.878409    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-workload-certs\") pod \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") "
Sep 10 15:56:02.878551 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.878439    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istiod-ca-cert\") pod \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\" (UID: \"fb08e393-d2da-47de-bab0-c7019b7e4ec8\") "
Sep 10 15:56:02.878551 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.878425    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-data" (OuterVolumeSpecName: "istio-data") pod "fb08e393-d2da-47de-bab0-c7019b7e4ec8" (UID: "fb08e393-d2da-47de-bab0-c7019b7e4ec8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:56:02.878551 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.878475    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fb08e393-d2da-47de-bab0-c7019b7e4ec8" (UID: "fb08e393-d2da-47de-bab0-c7019b7e4ec8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:56:02.878871 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.878607    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fb08e393-d2da-47de-bab0-c7019b7e4ec8" (UID: "fb08e393-d2da-47de-bab0-c7019b7e4ec8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:56:02.878871 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.878689    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fb08e393-d2da-47de-bab0-c7019b7e4ec8" (UID: "fb08e393-d2da-47de-bab0-c7019b7e4ec8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:56:02.878871 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.878852    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-workload-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:56:02.879210 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.878875    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-data\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:56:02.879210 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.878890    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-credential-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:56:02.879210 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.878905    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fb08e393-d2da-47de-bab0-c7019b7e4ec8" (UID: "fb08e393-d2da-47de-bab0-c7019b7e4ec8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:56:02.879210 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.878904    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-workload-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:56:02.880602 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.880565    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fb08e393-d2da-47de-bab0-c7019b7e4ec8" (UID: "fb08e393-d2da-47de-bab0-c7019b7e4ec8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:56:02.880714 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.880601    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb08e393-d2da-47de-bab0-c7019b7e4ec8-kube-api-access-sfjwq" (OuterVolumeSpecName: "kube-api-access-sfjwq") pod "fb08e393-d2da-47de-bab0-c7019b7e4ec8" (UID: "fb08e393-d2da-47de-bab0-c7019b7e4ec8"). InnerVolumeSpecName "kube-api-access-sfjwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:56:02.880714 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.880664    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fb08e393-d2da-47de-bab0-c7019b7e4ec8" (UID: "fb08e393-d2da-47de-bab0-c7019b7e4ec8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:56:02.880714 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.880698    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-token" (OuterVolumeSpecName: "istio-token") pod "fb08e393-d2da-47de-bab0-c7019b7e4ec8" (UID: "fb08e393-d2da-47de-bab0-c7019b7e4ec8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:56:02.980064 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.980027    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-podinfo\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:56:02.980064 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.980056    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfjwq\" (UniqueName: \"kubernetes.io/projected/fb08e393-d2da-47de-bab0-c7019b7e4ec8-kube-api-access-sfjwq\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:56:02.980064 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.980066    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-envoy\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:56:02.980064 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.980074    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istio-token\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:56:02.980346 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:02.980083    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb08e393-d2da-47de-bab0-c7019b7e4ec8-istiod-ca-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:56:03.030065 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:03.030029    2564 generic.go:356] "Generic (PLEG): container finished" podID="fb08e393-d2da-47de-bab0-c7019b7e4ec8" containerID="3a481b0079e8e2c04eba281829dd959094b6084d80b8b11a98f6aabbeb7f868c" exitCode=0
Sep 10 15:56:03.030259 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:03.030091    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"
Sep 10 15:56:03.030259 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:03.030108    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh" event={"ID":"fb08e393-d2da-47de-bab0-c7019b7e4ec8","Type":"ContainerDied","Data":"3a481b0079e8e2c04eba281829dd959094b6084d80b8b11a98f6aabbeb7f868c"}
Sep 10 15:56:03.030259 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:03.030180    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh" event={"ID":"fb08e393-d2da-47de-bab0-c7019b7e4ec8","Type":"ContainerDied","Data":"edc590fa199f6dd5328300947efe614cacfc72d093c2636fde16ce5360621ec7"}
Sep 10 15:56:03.030259 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:03.030197    2564 scope.go:117] "RemoveContainer" containerID="3a481b0079e8e2c04eba281829dd959094b6084d80b8b11a98f6aabbeb7f868c"
Sep 10 15:56:03.043114 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:03.043086    2564 scope.go:117] "RemoveContainer" containerID="d4a4c9687c0ad7b283610f0ff0b3cec2129f56c2cb08123baec6719e8ebfcb50"
Sep 10 15:56:03.053883 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:03.053858    2564 scope.go:117] "RemoveContainer" containerID="3a481b0079e8e2c04eba281829dd959094b6084d80b8b11a98f6aabbeb7f868c"
Sep 10 15:56:03.054290 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:56:03.054265    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a481b0079e8e2c04eba281829dd959094b6084d80b8b11a98f6aabbeb7f868c\": container with ID starting with 3a481b0079e8e2c04eba281829dd959094b6084d80b8b11a98f6aabbeb7f868c not found: ID does not exist" containerID="3a481b0079e8e2c04eba281829dd959094b6084d80b8b11a98f6aabbeb7f868c"
Sep 10 15:56:03.054409 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:03.054302    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a481b0079e8e2c04eba281829dd959094b6084d80b8b11a98f6aabbeb7f868c"} err="failed to get container status \"3a481b0079e8e2c04eba281829dd959094b6084d80b8b11a98f6aabbeb7f868c\": rpc error: code = NotFound desc = could not find container \"3a481b0079e8e2c04eba281829dd959094b6084d80b8b11a98f6aabbeb7f868c\": container with ID starting with 3a481b0079e8e2c04eba281829dd959094b6084d80b8b11a98f6aabbeb7f868c not found: ID does not exist"
Sep 10 15:56:03.054409 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:03.054329    2564 scope.go:117] "RemoveContainer" containerID="d4a4c9687c0ad7b283610f0ff0b3cec2129f56c2cb08123baec6719e8ebfcb50"
Sep 10 15:56:03.054690 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:56:03.054669    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a4c9687c0ad7b283610f0ff0b3cec2129f56c2cb08123baec6719e8ebfcb50\": container with ID starting with d4a4c9687c0ad7b283610f0ff0b3cec2129f56c2cb08123baec6719e8ebfcb50 not found: ID does not exist" containerID="d4a4c9687c0ad7b283610f0ff0b3cec2129f56c2cb08123baec6719e8ebfcb50"
Sep 10 15:56:03.054763 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:03.054697    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a4c9687c0ad7b283610f0ff0b3cec2129f56c2cb08123baec6719e8ebfcb50"} err="failed to get container status \"d4a4c9687c0ad7b283610f0ff0b3cec2129f56c2cb08123baec6719e8ebfcb50\": rpc error: code = NotFound desc = could not find container \"d4a4c9687c0ad7b283610f0ff0b3cec2129f56c2cb08123baec6719e8ebfcb50\": container with ID starting with d4a4c9687c0ad7b283610f0ff0b3cec2129f56c2cb08123baec6719e8ebfcb50 not found: ID does not exist"
Sep 10 15:56:03.068880 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:03.068854    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"]
Sep 10 15:56:03.076501 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:03.076469    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-384e8433-gw-openshift-default-557c74ddfb-mr5mh"]
Sep 10 15:56:03.673004 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:03.672970    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb08e393-d2da-47de-bab0-c7019b7e4ec8" path="/var/lib/kubelet/pods/fb08e393-d2da-47de-bab0-c7019b7e4ec8/volumes"
Sep 10 15:56:27.101039 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:27.100996    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dacd235-76b4p"]
Sep 10 15:56:27.101964 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:27.101707    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="654d6d0e-a57b-49ef-b48c-b98c66559655" containerName="revoke-keys"
Sep 10 15:56:27.101964 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:27.101734    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="654d6d0e-a57b-49ef-b48c-b98c66559655" containerName="revoke-keys"
Sep 10 15:56:27.101964 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:27.101771    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb08e393-d2da-47de-bab0-c7019b7e4ec8" containerName="istio-proxy"
Sep 10 15:56:27.101964 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:27.101780    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb08e393-d2da-47de-bab0-c7019b7e4ec8" containerName="istio-proxy"
Sep 10 15:56:27.101964 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:27.101799    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb08e393-d2da-47de-bab0-c7019b7e4ec8" containerName="istio-proxy"
Sep 10 15:56:27.101964 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:27.101808    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb08e393-d2da-47de-bab0-c7019b7e4ec8" containerName="istio-proxy"
Sep 10 15:56:27.101964 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:27.101927    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb08e393-d2da-47de-bab0-c7019b7e4ec8" containerName="istio-proxy"
Sep 10 15:56:27.101964 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:27.101938    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="654d6d0e-a57b-49ef-b48c-b98c66559655" containerName="revoke-keys"
Sep 10 15:56:27.101964 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:27.101948    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb08e393-d2da-47de-bab0-c7019b7e4ec8" containerName="istio-proxy"
Sep 10 15:56:27.105384 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:27.105362    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dacd235-76b4p"
Sep 10 15:56:27.108954 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:27.108930    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2wghg\""
Sep 10 15:56:27.116603 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:27.116575    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dacd235-76b4p"]
Sep 10 15:56:27.240007 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:27.239974    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dacd235-76b4p\" (UID: \"31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dacd235-76b4p"
Sep 10 15:56:27.341595 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:27.341552    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dacd235-76b4p\" (UID: \"31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dacd235-76b4p"
Sep 10 15:56:27.342219 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:27.342202    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2dacd235-76b4p\" (UID: \"31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dacd235-76b4p"
Sep 10 15:56:27.417340 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:27.417258    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dacd235-76b4p"
Sep 10 15:56:27.694894 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:27.694858    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dacd235-76b4p"]
Sep 10 15:56:27.698251 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:56:27.698216    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a0f15a_ca7e_4ef7_bcd1_8ef9237317c3.slice/crio-5c5e8ab442d8ac62c1b3ef65d85b353cd72a19bd99d502ea1d4010ba6370a6bc WatchSource:0}: Error finding container 5c5e8ab442d8ac62c1b3ef65d85b353cd72a19bd99d502ea1d4010ba6370a6bc: Status 404 returned error can't find the container with id 5c5e8ab442d8ac62c1b3ef65d85b353cd72a19bd99d502ea1d4010ba6370a6bc
Sep 10 15:56:28.179343 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:28.179303    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dacd235-76b4p" event={"ID":"31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3","Type":"ContainerStarted","Data":"d87457ea7e7635599ad385975c9923c8dcf775a4297c0287a6f734cfb20da861"}
Sep 10 15:56:28.179799 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:28.179387    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dacd235-76b4p" event={"ID":"31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3","Type":"ContainerStarted","Data":"5c5e8ab442d8ac62c1b3ef65d85b353cd72a19bd99d502ea1d4010ba6370a6bc"}
Sep 10 15:56:28.205674 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:28.205603    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dacd235-76b4p" podStartSLOduration=1.205580625 podStartE2EDuration="1.205580625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:56:28.2033292 +0000 UTC m=+867.007648461" watchObservedRunningTime="2026-09-10 15:56:28.205580625 +0000 UTC m=+867.009899885"
Sep 10 15:56:29.185435 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:29.185397    2564 generic.go:356] "Generic (PLEG): container finished" podID="31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3" containerID="d87457ea7e7635599ad385975c9923c8dcf775a4297c0287a6f734cfb20da861" exitCode=0
Sep 10 15:56:29.185824 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:29.185477    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dacd235-76b4p" event={"ID":"31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3","Type":"ContainerDied","Data":"d87457ea7e7635599ad385975c9923c8dcf775a4297c0287a6f734cfb20da861"}
Sep 10 15:56:30.336479 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:30.336451    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dacd235-76b4p"
Sep 10 15:56:30.474964 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:30.474868    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3-maas-api-service-ca\") pod \"31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3\" (UID: \"31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3\") "
Sep 10 15:56:30.475272 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:30.475248    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3" (UID: "31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:56:30.576262 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:30.576226    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3-maas-api-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:56:31.198336 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:31.198306    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dacd235-76b4p"
Sep 10 15:56:31.198513 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:31.198335    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dacd235-76b4p" event={"ID":"31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3","Type":"ContainerDied","Data":"5c5e8ab442d8ac62c1b3ef65d85b353cd72a19bd99d502ea1d4010ba6370a6bc"}
Sep 10 15:56:31.198513 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:31.198391    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c5e8ab442d8ac62c1b3ef65d85b353cd72a19bd99d502ea1d4010ba6370a6bc"
Sep 10 15:56:33.740873 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:33.740842    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dacd235-76b4p"]
Sep 10 15:56:33.744478 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:33.744454    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2dacd235-76b4p"]
Sep 10 15:56:35.673224 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:35.673191    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3" path="/var/lib/kubelet/pods/31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3/volumes"
Sep 10 15:56:36.071528 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.071452    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"]
Sep 10 15:56:36.072101 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.072073    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3" containerName="revoke-keys"
Sep 10 15:56:36.072101 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.072101    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3" containerName="revoke-keys"
Sep 10 15:56:36.072332 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.072243    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="31a0f15a-ca7e-4ef7-bcd1-8ef9237317c3" containerName="revoke-keys"
Sep 10 15:56:36.075653 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.075625    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.078234 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.078214    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-8d528235-openshift-default-dockercfg-8zpcv\""
Sep 10 15:56:36.105452 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.105422    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"]
Sep 10 15:56:36.139800 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.139764    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3f5304-6120-4ec9-bc11-819124adc58c-istiod-ca-cert\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.140014 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.139836    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzdq7\" (UniqueName: \"kubernetes.io/projected/1c3f5304-6120-4ec9-bc11-819124adc58c-kube-api-access-zzdq7\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.140014 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.139858    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-podinfo\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.140014 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.139885    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-credential-socket\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.140014 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.139907    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-token\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.140014 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.139957    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-workload-certs\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.140014 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.139998    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-data\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.140426 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.140042    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-envoy\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.140426 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.140087    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-workload-socket\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.240951 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.240918    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-envoy\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.241137 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.240984    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-workload-socket\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.241137 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.241013    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3f5304-6120-4ec9-bc11-819124adc58c-istiod-ca-cert\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.241298 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.241146    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzdq7\" (UniqueName: \"kubernetes.io/projected/1c3f5304-6120-4ec9-bc11-819124adc58c-kube-api-access-zzdq7\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.241298 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.241186    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-podinfo\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.241298 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.241216    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-credential-socket\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.241298 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.241233    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-token\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.241298 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.241266    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-workload-certs\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.241575 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.241303    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-data\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.241575 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.241415    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-workload-socket\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.241575 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.241566    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-credential-socket\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.241745 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.241618    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-data\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.241850 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.241823    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3f5304-6120-4ec9-bc11-819124adc58c-istiod-ca-cert\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.242041 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.242021    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-workload-certs\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.243476 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.243457    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-envoy\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.244005 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.243987    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-podinfo\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.258716 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.258678    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzdq7\" (UniqueName: \"kubernetes.io/projected/1c3f5304-6120-4ec9-bc11-819124adc58c-kube-api-access-zzdq7\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.271953 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.271926    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-token\") pod \"e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") " pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.386931 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.386849    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:36.680384 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.680358    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"]
Sep 10 15:56:36.681695 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:56:36.681671    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3f5304_6120_4ec9_bc11_819124adc58c.slice/crio-8035ab56753683895cb28587e96547ed98e10e00848ea6a5fe632c23bbe72a7d WatchSource:0}: Error finding container 8035ab56753683895cb28587e96547ed98e10e00848ea6a5fe632c23bbe72a7d: Status 404 returned error can't find the container with id 8035ab56753683895cb28587e96547ed98e10e00848ea6a5fe632c23bbe72a7d
Sep 10 15:56:36.683963 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.683929    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:56:36.684043 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.683997    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:56:36.684043 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:36.684030    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:56:37.235678 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:37.235647    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv" event={"ID":"1c3f5304-6120-4ec9-bc11-819124adc58c","Type":"ContainerStarted","Data":"47abb231c4baa45ff84f9263f555d538a01a0bf9a296d449294e89678fcec8e4"}
Sep 10 15:56:37.235868 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:37.235698    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv" event={"ID":"1c3f5304-6120-4ec9-bc11-819124adc58c","Type":"ContainerStarted","Data":"8035ab56753683895cb28587e96547ed98e10e00848ea6a5fe632c23bbe72a7d"}
Sep 10 15:56:37.267152 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:37.267099    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv" podStartSLOduration=1.267080834 podStartE2EDuration="1.267080834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:56:37.266371532 +0000 UTC m=+876.070690794" watchObservedRunningTime="2026-09-10 15:56:37.267080834 +0000 UTC m=+876.071400095"
Sep 10 15:56:37.387783 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:37.387749    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:37.393520 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:37.393482    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:38.241491 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:38.241457    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:38.242628 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:38.242607    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:57.451907 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.451871    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"]
Sep 10 15:56:57.455677 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.455654    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:56:57.459793 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.459765    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-87a024\"/\"autores-model-87a024-kserve-self-signed-certs\""
Sep 10 15:56:57.460038 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.459766    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-87a024\"/\"kube-root-ca.crt\""
Sep 10 15:56:57.460211 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.459817    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-87a024\"/\"openshift-service-ca.crt\""
Sep 10 15:56:57.460540 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.459827    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-87a024\"/\"default-dockercfg-7vrzf\""
Sep 10 15:56:57.467061 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.467032    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"]
Sep 10 15:56:57.573609 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.573575    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-dshm\") pod \"autores-model-87a024-kserve-6bc5f968c5-m7wj5\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") " pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:56:57.573609 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.573613    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01913550-0fd3-404e-bc81-8fab500c4f62-tls-certs\") pod \"autores-model-87a024-kserve-6bc5f968c5-m7wj5\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") " pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:56:57.573816 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.573635    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-tmp-dir\") pod \"autores-model-87a024-kserve-6bc5f968c5-m7wj5\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") " pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:56:57.573816 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.573708    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-model-cache\") pod \"autores-model-87a024-kserve-6bc5f968c5-m7wj5\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") " pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:56:57.573816 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.573760    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwptr\" (UniqueName: \"kubernetes.io/projected/01913550-0fd3-404e-bc81-8fab500c4f62-kube-api-access-jwptr\") pod \"autores-model-87a024-kserve-6bc5f968c5-m7wj5\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") " pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:56:57.573816 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.573791    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-home\") pod \"autores-model-87a024-kserve-6bc5f968c5-m7wj5\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") " pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:56:57.674972 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.674943    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwptr\" (UniqueName: \"kubernetes.io/projected/01913550-0fd3-404e-bc81-8fab500c4f62-kube-api-access-jwptr\") pod \"autores-model-87a024-kserve-6bc5f968c5-m7wj5\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") " pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:56:57.675224 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.674980    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-home\") pod \"autores-model-87a024-kserve-6bc5f968c5-m7wj5\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") " pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:56:57.675224 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.675068    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-dshm\") pod \"autores-model-87a024-kserve-6bc5f968c5-m7wj5\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") " pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:56:57.675224 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.675095    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01913550-0fd3-404e-bc81-8fab500c4f62-tls-certs\") pod \"autores-model-87a024-kserve-6bc5f968c5-m7wj5\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") " pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:56:57.675224 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.675120    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-tmp-dir\") pod \"autores-model-87a024-kserve-6bc5f968c5-m7wj5\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") " pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:56:57.675224 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.675189    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-model-cache\") pod \"autores-model-87a024-kserve-6bc5f968c5-m7wj5\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") " pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:56:57.675577 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.675548    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-home\") pod \"autores-model-87a024-kserve-6bc5f968c5-m7wj5\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") " pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:56:57.675653 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.675574    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-tmp-dir\") pod \"autores-model-87a024-kserve-6bc5f968c5-m7wj5\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") " pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:56:57.675653 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.675613    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-model-cache\") pod \"autores-model-87a024-kserve-6bc5f968c5-m7wj5\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") " pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:56:57.677511 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.677488    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-dshm\") pod \"autores-model-87a024-kserve-6bc5f968c5-m7wj5\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") " pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:56:57.677710 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.677696    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01913550-0fd3-404e-bc81-8fab500c4f62-tls-certs\") pod \"autores-model-87a024-kserve-6bc5f968c5-m7wj5\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") " pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:56:57.685466 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.685439    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwptr\" (UniqueName: \"kubernetes.io/projected/01913550-0fd3-404e-bc81-8fab500c4f62-kube-api-access-jwptr\") pod \"autores-model-87a024-kserve-6bc5f968c5-m7wj5\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") " pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:56:57.769715 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:57.769621    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:56:58.065273 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:58.065245    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"]
Sep 10 15:56:58.067690 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:56:58.067663    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01913550_0fd3_404e_bc81_8fab500c4f62.slice/crio-68332130d6398939ae91a87373d481c9e79c5af684924d28a9d5449a5275a2bc WatchSource:0}: Error finding container 68332130d6398939ae91a87373d481c9e79c5af684924d28a9d5449a5275a2bc: Status 404 returned error can't find the container with id 68332130d6398939ae91a87373d481c9e79c5af684924d28a9d5449a5275a2bc
Sep 10 15:56:58.372690 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:58.372595    2564 generic.go:356] "Generic (PLEG): container finished" podID="1c3f5304-6120-4ec9-bc11-819124adc58c" containerID="47abb231c4baa45ff84f9263f555d538a01a0bf9a296d449294e89678fcec8e4" exitCode=0
Sep 10 15:56:58.372863 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:58.372677    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv" event={"ID":"1c3f5304-6120-4ec9-bc11-819124adc58c","Type":"ContainerDied","Data":"47abb231c4baa45ff84f9263f555d538a01a0bf9a296d449294e89678fcec8e4"}
Sep 10 15:56:58.373213 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:58.373195    2564 scope.go:117] "RemoveContainer" containerID="47abb231c4baa45ff84f9263f555d538a01a0bf9a296d449294e89678fcec8e4"
Sep 10 15:56:58.374320 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:58.374287    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5" event={"ID":"01913550-0fd3-404e-bc81-8fab500c4f62","Type":"ContainerStarted","Data":"e3c4b6617e9c9f9bb54fbb8eb3704d7fd4ecc2855dc4368661358c2b23bb8b2d"}
Sep 10 15:56:58.374410 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:58.374324    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5" event={"ID":"01913550-0fd3-404e-bc81-8fab500c4f62","Type":"ContainerStarted","Data":"68332130d6398939ae91a87373d481c9e79c5af684924d28a9d5449a5275a2bc"}
Sep 10 15:56:58.374472 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:58.374444    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:56:58.374536 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:58.374522    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:56:58.374578 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:58.374565    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:56:58.427380 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:58.427329    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5" podStartSLOduration=1.427312352 podStartE2EDuration="1.427312352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:56:58.424588628 +0000 UTC m=+897.228907890" watchObservedRunningTime="2026-09-10 15:56:58.427312352 +0000 UTC m=+897.231631676"
Sep 10 15:56:59.381593 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:59.381551    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv" event={"ID":"1c3f5304-6120-4ec9-bc11-819124adc58c","Type":"ContainerStarted","Data":"0cd3533943ac8768459a11d60b74fbd6c34a03a0eb5cc01345ca3399595c8a62"}
Sep 10 15:56:59.387146 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:59.387120    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:56:59.387375 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:59.387353    2564 patch_prober.go:28] interesting pod/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 10 15:56:59.387432 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:56:59.387407    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv" podUID="1c3f5304-6120-4ec9-bc11-819124adc58c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 10 15:57:00.387377 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:00.387340    2564 patch_prober.go:28] interesting pod/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 10 15:57:00.387858 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:00.387400    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv" podUID="1c3f5304-6120-4ec9-bc11-819124adc58c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 10 15:57:01.388213 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:01.388152    2564 patch_prober.go:28] interesting pod/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 10 15:57:01.388661 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:01.388245    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv" podUID="1c3f5304-6120-4ec9-bc11-819124adc58c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 10 15:57:02.356543 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:02.356509    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-snm4z_f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f/console-operator/1.log"
Sep 10 15:57:02.358090 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:02.358066    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-snm4z_f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f/console-operator/1.log"
Sep 10 15:57:02.387970 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:02.387596    2564 patch_prober.go:28] interesting pod/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 10 15:57:02.387970 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:02.387647    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv" podUID="1c3f5304-6120-4ec9-bc11-819124adc58c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 10 15:57:03.387896 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:03.387855    2564 patch_prober.go:28] interesting pod/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 10 15:57:03.388323 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:03.387928    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv" podUID="1c3f5304-6120-4ec9-bc11-819124adc58c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 10 15:57:04.387970 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:04.387931    2564 patch_prober.go:28] interesting pod/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 10 15:57:04.388424 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:04.387989    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv" podUID="1c3f5304-6120-4ec9-bc11-819124adc58c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 10 15:57:05.391897 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:05.391863    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:57:05.392384 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:05.392146    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:57:05.392975 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:05.392959    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:57:07.770302 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:07.770260    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:57:07.770302 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:07.770305    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:57:07.783046 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:07.783016    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:57:08.392609 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.392581    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"]
Sep 10 15:57:08.395987 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.395970    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:08.401920 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.401672    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-83dcaf\"/\"default-dockercfg-pszcl\""
Sep 10 15:57:08.401920 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.401739    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-83dcaf\"/\"kube-root-ca.crt\""
Sep 10 15:57:08.401920 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.401749    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-83dcaf\"/\"openshift-service-ca.crt\""
Sep 10 15:57:08.401920 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.401768    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-83dcaf\"/\"test-model-83dcaf-kserve-self-signed-certs\""
Sep 10 15:57:08.425455 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.425424    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"]
Sep 10 15:57:08.474643 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.474611    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:57:08.504762 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.504733    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-home\") pod \"test-model-83dcaf-kserve-788df77fff-7sc5b\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") " pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:08.504938 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.504773    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-tmp-dir\") pod \"test-model-83dcaf-kserve-788df77fff-7sc5b\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") " pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:08.504938 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.504799    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de68757c-acc3-4495-b69b-4e05cff36df9-tls-certs\") pod \"test-model-83dcaf-kserve-788df77fff-7sc5b\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") " pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:08.504938 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.504853    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-model-cache\") pod \"test-model-83dcaf-kserve-788df77fff-7sc5b\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") " pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:08.505092 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.504984    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-dshm\") pod \"test-model-83dcaf-kserve-788df77fff-7sc5b\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") " pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:08.505092 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.505013    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94qtx\" (UniqueName: \"kubernetes.io/projected/de68757c-acc3-4495-b69b-4e05cff36df9-kube-api-access-94qtx\") pod \"test-model-83dcaf-kserve-788df77fff-7sc5b\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") " pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:08.605835 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.605799    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-dshm\") pod \"test-model-83dcaf-kserve-788df77fff-7sc5b\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") " pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:08.605835 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.605836    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94qtx\" (UniqueName: \"kubernetes.io/projected/de68757c-acc3-4495-b69b-4e05cff36df9-kube-api-access-94qtx\") pod \"test-model-83dcaf-kserve-788df77fff-7sc5b\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") " pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:08.606087 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.606011    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-home\") pod \"test-model-83dcaf-kserve-788df77fff-7sc5b\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") " pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:08.606087 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.606066    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-tmp-dir\") pod \"test-model-83dcaf-kserve-788df77fff-7sc5b\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") " pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:08.606240 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.606125    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de68757c-acc3-4495-b69b-4e05cff36df9-tls-certs\") pod \"test-model-83dcaf-kserve-788df77fff-7sc5b\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") " pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:08.606240 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.606204    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-model-cache\") pod \"test-model-83dcaf-kserve-788df77fff-7sc5b\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") " pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:08.606464 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.606439    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-home\") pod \"test-model-83dcaf-kserve-788df77fff-7sc5b\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") " pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:08.606553 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.606461    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-tmp-dir\") pod \"test-model-83dcaf-kserve-788df77fff-7sc5b\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") " pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:08.606799 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.606779    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-model-cache\") pod \"test-model-83dcaf-kserve-788df77fff-7sc5b\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") " pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:08.608149 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.608134    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-dshm\") pod \"test-model-83dcaf-kserve-788df77fff-7sc5b\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") " pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:08.608710 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.608688    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de68757c-acc3-4495-b69b-4e05cff36df9-tls-certs\") pod \"test-model-83dcaf-kserve-788df77fff-7sc5b\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") " pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:08.618012 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.617990    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94qtx\" (UniqueName: \"kubernetes.io/projected/de68757c-acc3-4495-b69b-4e05cff36df9-kube-api-access-94qtx\") pod \"test-model-83dcaf-kserve-788df77fff-7sc5b\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") " pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:08.673475 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.673446    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d528235-wl5lt"]
Sep 10 15:57:08.676615 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.676599    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d528235-wl5lt"
Sep 10 15:57:08.679688 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.679668    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2wghg\""
Sep 10 15:57:08.687029 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.686944    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d528235-wl5lt"]
Sep 10 15:57:08.706616 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.706579    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:08.808507 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.808412    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b97534-0d68-42d2-b1b0-aeadc53b921e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-8d528235-wl5lt\" (UID: \"60b97534-0d68-42d2-b1b0-aeadc53b921e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d528235-wl5lt"
Sep 10 15:57:08.909971 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.909933    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b97534-0d68-42d2-b1b0-aeadc53b921e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-8d528235-wl5lt\" (UID: \"60b97534-0d68-42d2-b1b0-aeadc53b921e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d528235-wl5lt"
Sep 10 15:57:08.910812 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.910785    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b97534-0d68-42d2-b1b0-aeadc53b921e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-8d528235-wl5lt\" (UID: \"60b97534-0d68-42d2-b1b0-aeadc53b921e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d528235-wl5lt"
Sep 10 15:57:08.992193 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:08.992067    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d528235-wl5lt"
Sep 10 15:57:09.001477 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:09.001436    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"]
Sep 10 15:57:09.005653 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:57:09.005619    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde68757c_acc3_4495_b69b_4e05cff36df9.slice/crio-7194839c2082727d4c16977c00bba7775a9ae7d7743d9b8bcebbd51df2b69a1c WatchSource:0}: Error finding container 7194839c2082727d4c16977c00bba7775a9ae7d7743d9b8bcebbd51df2b69a1c: Status 404 returned error can't find the container with id 7194839c2082727d4c16977c00bba7775a9ae7d7743d9b8bcebbd51df2b69a1c
Sep 10 15:57:09.286492 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:09.286412    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d528235-wl5lt"]
Sep 10 15:57:09.289592 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:57:09.289552    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b97534_0d68_42d2_b1b0_aeadc53b921e.slice/crio-67028e08c73830672ced69ddecf24a80cbe6822deac7c61225ad8a0a82d976a1 WatchSource:0}: Error finding container 67028e08c73830672ced69ddecf24a80cbe6822deac7c61225ad8a0a82d976a1: Status 404 returned error can't find the container with id 67028e08c73830672ced69ddecf24a80cbe6822deac7c61225ad8a0a82d976a1
Sep 10 15:57:09.469900 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:09.469868    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d528235-wl5lt" event={"ID":"60b97534-0d68-42d2-b1b0-aeadc53b921e","Type":"ContainerStarted","Data":"67028e08c73830672ced69ddecf24a80cbe6822deac7c61225ad8a0a82d976a1"}
Sep 10 15:57:09.471393 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:09.471363    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b" event={"ID":"de68757c-acc3-4495-b69b-4e05cff36df9","Type":"ContainerStarted","Data":"52dae10ccfd7307feadac9410f176f469d8e471dc6bc8a4c12ba9e22ab4877f9"}
Sep 10 15:57:09.471534 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:09.471401    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b" event={"ID":"de68757c-acc3-4495-b69b-4e05cff36df9","Type":"ContainerStarted","Data":"7194839c2082727d4c16977c00bba7775a9ae7d7743d9b8bcebbd51df2b69a1c"}
Sep 10 15:57:09.494186 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:09.494120    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b" podStartSLOduration=1.49410218 podStartE2EDuration="1.49410218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:57:09.491364732 +0000 UTC m=+908.295684006" watchObservedRunningTime="2026-09-10 15:57:09.49410218 +0000 UTC m=+908.298421431"
Sep 10 15:57:10.479471 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:10.479381    2564 generic.go:356] "Generic (PLEG): container finished" podID="60b97534-0d68-42d2-b1b0-aeadc53b921e" containerID="deb47ec555ac6212fc1ca9563257fd574fe6720d2131bae065793b1d4d379039" exitCode=0
Sep 10 15:57:10.479471 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:10.479434    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d528235-wl5lt" event={"ID":"60b97534-0d68-42d2-b1b0-aeadc53b921e","Type":"ContainerDied","Data":"deb47ec555ac6212fc1ca9563257fd574fe6720d2131bae065793b1d4d379039"}
Sep 10 15:57:11.637025 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:11.636995    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d528235-wl5lt"
Sep 10 15:57:11.739878 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:11.739787    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b97534-0d68-42d2-b1b0-aeadc53b921e-maas-api-service-ca\") pod \"60b97534-0d68-42d2-b1b0-aeadc53b921e\" (UID: \"60b97534-0d68-42d2-b1b0-aeadc53b921e\") "
Sep 10 15:57:11.740220 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:11.740196    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b97534-0d68-42d2-b1b0-aeadc53b921e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60b97534-0d68-42d2-b1b0-aeadc53b921e" (UID: "60b97534-0d68-42d2-b1b0-aeadc53b921e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:57:11.841260 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:11.841226    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b97534-0d68-42d2-b1b0-aeadc53b921e-maas-api-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:12.491963 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:12.491932    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d528235-wl5lt"
Sep 10 15:57:12.492299 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:12.491935    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d528235-wl5lt" event={"ID":"60b97534-0d68-42d2-b1b0-aeadc53b921e","Type":"ContainerDied","Data":"67028e08c73830672ced69ddecf24a80cbe6822deac7c61225ad8a0a82d976a1"}
Sep 10 15:57:12.492437 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:12.492360    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67028e08c73830672ced69ddecf24a80cbe6822deac7c61225ad8a0a82d976a1"
Sep 10 15:57:13.673678 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:13.673648    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"]
Sep 10 15:57:13.816385 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:13.816355    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-87a024-h8s8x"]
Sep 10 15:57:13.816874 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:13.816861    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60b97534-0d68-42d2-b1b0-aeadc53b921e" containerName="revoke-keys"
Sep 10 15:57:13.816920 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:13.816877    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="60b97534-0d68-42d2-b1b0-aeadc53b921e" containerName="revoke-keys"
Sep 10 15:57:13.816960 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:13.816951    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="60b97534-0d68-42d2-b1b0-aeadc53b921e" containerName="revoke-keys"
Sep 10 15:57:13.820318 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:13.820291    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-87a024-h8s8x"
Sep 10 15:57:13.823120 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:13.823097    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2wghg\""
Sep 10 15:57:13.835081 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:13.835054    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-87a024-h8s8x"]
Sep 10 15:57:13.969444 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:13.969353    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55016ec4-918c-4f80-bd6b-e8f2b8a99b01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-87a024-h8s8x\" (UID: \"55016ec4-918c-4f80-bd6b-e8f2b8a99b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-87a024-h8s8x"
Sep 10 15:57:14.070256 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:14.070220    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55016ec4-918c-4f80-bd6b-e8f2b8a99b01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-87a024-h8s8x\" (UID: \"55016ec4-918c-4f80-bd6b-e8f2b8a99b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-87a024-h8s8x"
Sep 10 15:57:14.070856 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:14.070835    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55016ec4-918c-4f80-bd6b-e8f2b8a99b01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-87a024-h8s8x\" (UID: \"55016ec4-918c-4f80-bd6b-e8f2b8a99b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-87a024-h8s8x"
Sep 10 15:57:14.132653 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:14.132622    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-87a024-h8s8x"
Sep 10 15:57:14.413480 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:14.413338    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-87a024-h8s8x"]
Sep 10 15:57:14.416405 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:57:14.416375    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55016ec4_918c_4f80_bd6b_e8f2b8a99b01.slice/crio-35b82881b91b3e0c900c22ed43b3bd6db60289e4cd0df92430bdfe3cc5ae35cd WatchSource:0}: Error finding container 35b82881b91b3e0c900c22ed43b3bd6db60289e4cd0df92430bdfe3cc5ae35cd: Status 404 returned error can't find the container with id 35b82881b91b3e0c900c22ed43b3bd6db60289e4cd0df92430bdfe3cc5ae35cd
Sep 10 15:57:14.507221 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:14.507182    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-87a024-h8s8x" event={"ID":"55016ec4-918c-4f80-bd6b-e8f2b8a99b01","Type":"ContainerStarted","Data":"35b82881b91b3e0c900c22ed43b3bd6db60289e4cd0df92430bdfe3cc5ae35cd"}
Sep 10 15:57:15.514103 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:15.514067    2564 generic.go:356] "Generic (PLEG): container finished" podID="55016ec4-918c-4f80-bd6b-e8f2b8a99b01" containerID="f0e2212dcc0530c25b8e3ed5dd293d3e932c08b39f557f320396628eedb7dd92" exitCode=0
Sep 10 15:57:15.514495 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:15.514118    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-87a024-h8s8x" event={"ID":"55016ec4-918c-4f80-bd6b-e8f2b8a99b01","Type":"ContainerDied","Data":"f0e2212dcc0530c25b8e3ed5dd293d3e932c08b39f557f320396628eedb7dd92"}
Sep 10 15:57:16.675385 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:16.675345    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-87a024-h8s8x"
Sep 10 15:57:16.800565 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:16.800455    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55016ec4-918c-4f80-bd6b-e8f2b8a99b01-maas-api-service-ca\") pod \"55016ec4-918c-4f80-bd6b-e8f2b8a99b01\" (UID: \"55016ec4-918c-4f80-bd6b-e8f2b8a99b01\") "
Sep 10 15:57:16.801434 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:16.801406    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55016ec4-918c-4f80-bd6b-e8f2b8a99b01-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "55016ec4-918c-4f80-bd6b-e8f2b8a99b01" (UID: "55016ec4-918c-4f80-bd6b-e8f2b8a99b01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:57:16.902402 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:16.902363    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55016ec4-918c-4f80-bd6b-e8f2b8a99b01-maas-api-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:17.530042 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:17.529997    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-87a024-h8s8x" event={"ID":"55016ec4-918c-4f80-bd6b-e8f2b8a99b01","Type":"ContainerDied","Data":"35b82881b91b3e0c900c22ed43b3bd6db60289e4cd0df92430bdfe3cc5ae35cd"}
Sep 10 15:57:17.530279 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:17.530043    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-87a024-h8s8x"
Sep 10 15:57:17.530279 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:17.530062    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35b82881b91b3e0c900c22ed43b3bd6db60289e4cd0df92430bdfe3cc5ae35cd"
Sep 10 15:57:18.707797 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:18.707759    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:18.708297 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:18.707819    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:18.720719 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:18.720692    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:19.553833 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:19.553804    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:57:21.508790 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:21.508747    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d528235-wl5lt"]
Sep 10 15:57:21.514180 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:21.514140    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8d528235-wl5lt"]
Sep 10 15:57:21.679349 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:21.679301    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60b97534-0d68-42d2-b1b0-aeadc53b921e" path="/var/lib/kubelet/pods/60b97534-0d68-42d2-b1b0-aeadc53b921e/volumes"
Sep 10 15:57:21.692424 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:21.692392    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"]
Sep 10 15:57:21.692638 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:21.692611    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv" podUID="1c3f5304-6120-4ec9-bc11-819124adc58c" containerName="istio-proxy" containerID="cri-o://0cd3533943ac8768459a11d60b74fbd6c34a03a0eb5cc01345ca3399595c8a62" gracePeriod=30
Sep 10 15:57:24.744203 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:24.744145    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm"]
Sep 10 15:57:24.744721 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:24.744426    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm" podUID="0fc405ea-b5f4-4c99-ba4c-bc98aac499ea" containerName="payload-processing" containerID="cri-o://4fd0abac3fbb4c3c7730a3bd088979a1642e1579b6dea8cc6f21678fe280c762" gracePeriod=30
Sep 10 15:57:25.017436 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:25.017339    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-87a024-h8s8x"]
Sep 10 15:57:25.019683 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:25.019657    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-87a024-h8s8x"]
Sep 10 15:57:25.043186 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:25.043148    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm"
Sep 10 15:57:25.083736 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:25.083702    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqhhd\" (UniqueName: \"kubernetes.io/projected/0fc405ea-b5f4-4c99-ba4c-bc98aac499ea-kube-api-access-hqhhd\") pod \"0fc405ea-b5f4-4c99-ba4c-bc98aac499ea\" (UID: \"0fc405ea-b5f4-4c99-ba4c-bc98aac499ea\") "
Sep 10 15:57:25.083904 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:25.083854    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc405ea-b5f4-4c99-ba4c-bc98aac499ea-plugins-config-volume\") pod \"0fc405ea-b5f4-4c99-ba4c-bc98aac499ea\" (UID: \"0fc405ea-b5f4-4c99-ba4c-bc98aac499ea\") "
Sep 10 15:57:25.084251 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:25.084227    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc405ea-b5f4-4c99-ba4c-bc98aac499ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0fc405ea-b5f4-4c99-ba4c-bc98aac499ea" (UID: "0fc405ea-b5f4-4c99-ba4c-bc98aac499ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:57:25.086045 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:25.086018    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc405ea-b5f4-4c99-ba4c-bc98aac499ea-kube-api-access-hqhhd" (OuterVolumeSpecName: "kube-api-access-hqhhd") pod "0fc405ea-b5f4-4c99-ba4c-bc98aac499ea" (UID: "0fc405ea-b5f4-4c99-ba4c-bc98aac499ea"). InnerVolumeSpecName "kube-api-access-hqhhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:57:25.185359 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:25.185312    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqhhd\" (UniqueName: \"kubernetes.io/projected/0fc405ea-b5f4-4c99-ba4c-bc98aac499ea-kube-api-access-hqhhd\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:25.185359 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:25.185349    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc405ea-b5f4-4c99-ba4c-bc98aac499ea-plugins-config-volume\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:25.577221 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:25.577186    2564 generic.go:356] "Generic (PLEG): container finished" podID="0fc405ea-b5f4-4c99-ba4c-bc98aac499ea" containerID="4fd0abac3fbb4c3c7730a3bd088979a1642e1579b6dea8cc6f21678fe280c762" exitCode=0
Sep 10 15:57:25.577402 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:25.577235    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm"
Sep 10 15:57:25.577402 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:25.577242    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm" event={"ID":"0fc405ea-b5f4-4c99-ba4c-bc98aac499ea","Type":"ContainerDied","Data":"4fd0abac3fbb4c3c7730a3bd088979a1642e1579b6dea8cc6f21678fe280c762"}
Sep 10 15:57:25.577402 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:25.577283    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm" event={"ID":"0fc405ea-b5f4-4c99-ba4c-bc98aac499ea","Type":"ContainerDied","Data":"3f36ada16da06678a9f66515e665a913c4e4abaf1f812100bf20e266275c563d"}
Sep 10 15:57:25.577402 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:25.577303    2564 scope.go:117] "RemoveContainer" containerID="4fd0abac3fbb4c3c7730a3bd088979a1642e1579b6dea8cc6f21678fe280c762"
Sep 10 15:57:25.586903 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:25.586885    2564 scope.go:117] "RemoveContainer" containerID="4fd0abac3fbb4c3c7730a3bd088979a1642e1579b6dea8cc6f21678fe280c762"
Sep 10 15:57:25.587203 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:57:25.587182    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd0abac3fbb4c3c7730a3bd088979a1642e1579b6dea8cc6f21678fe280c762\": container with ID starting with 4fd0abac3fbb4c3c7730a3bd088979a1642e1579b6dea8cc6f21678fe280c762 not found: ID does not exist" containerID="4fd0abac3fbb4c3c7730a3bd088979a1642e1579b6dea8cc6f21678fe280c762"
Sep 10 15:57:25.587256 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:25.587213    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd0abac3fbb4c3c7730a3bd088979a1642e1579b6dea8cc6f21678fe280c762"} err="failed to get container status \"4fd0abac3fbb4c3c7730a3bd088979a1642e1579b6dea8cc6f21678fe280c762\": rpc error: code = NotFound desc = could not find container \"4fd0abac3fbb4c3c7730a3bd088979a1642e1579b6dea8cc6f21678fe280c762\": container with ID starting with 4fd0abac3fbb4c3c7730a3bd088979a1642e1579b6dea8cc6f21678fe280c762 not found: ID does not exist"
Sep 10 15:57:25.603612 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:25.603580    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm"]
Sep 10 15:57:25.608961 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:25.608936    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-87a024-6bd4c565-bwglm"]
Sep 10 15:57:25.672750 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:25.672720    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fc405ea-b5f4-4c99-ba4c-bc98aac499ea" path="/var/lib/kubelet/pods/0fc405ea-b5f4-4c99-ba4c-bc98aac499ea/volumes"
Sep 10 15:57:25.673078 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:25.673064    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55016ec4-918c-4f80-bd6b-e8f2b8a99b01" path="/var/lib/kubelet/pods/55016ec4-918c-4f80-bd6b-e8f2b8a99b01/volumes"
Sep 10 15:57:26.963569 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:26.963502    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:57:27.003737 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.003712    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-credential-socket\") pod \"1c3f5304-6120-4ec9-bc11-819124adc58c\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") "
Sep 10 15:57:27.003890 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.003748    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-workload-certs\") pod \"1c3f5304-6120-4ec9-bc11-819124adc58c\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") "
Sep 10 15:57:27.003890 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.003770    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-envoy\") pod \"1c3f5304-6120-4ec9-bc11-819124adc58c\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") "
Sep 10 15:57:27.003890 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.003803    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-podinfo\") pod \"1c3f5304-6120-4ec9-bc11-819124adc58c\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") "
Sep 10 15:57:27.003890 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.003850    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3f5304-6120-4ec9-bc11-819124adc58c-istiod-ca-cert\") pod \"1c3f5304-6120-4ec9-bc11-819124adc58c\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") "
Sep 10 15:57:27.003890 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.003883    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-workload-socket\") pod \"1c3f5304-6120-4ec9-bc11-819124adc58c\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") "
Sep 10 15:57:27.004205 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.003998    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzdq7\" (UniqueName: \"kubernetes.io/projected/1c3f5304-6120-4ec9-bc11-819124adc58c-kube-api-access-zzdq7\") pod \"1c3f5304-6120-4ec9-bc11-819124adc58c\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") "
Sep 10 15:57:27.004205 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.004028    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1c3f5304-6120-4ec9-bc11-819124adc58c" (UID: "1c3f5304-6120-4ec9-bc11-819124adc58c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:27.004205 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.004039    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1c3f5304-6120-4ec9-bc11-819124adc58c" (UID: "1c3f5304-6120-4ec9-bc11-819124adc58c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:27.004205 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.004083    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-data\") pod \"1c3f5304-6120-4ec9-bc11-819124adc58c\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") "
Sep 10 15:57:27.004205 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.004118    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-token\") pod \"1c3f5304-6120-4ec9-bc11-819124adc58c\" (UID: \"1c3f5304-6120-4ec9-bc11-819124adc58c\") "
Sep 10 15:57:27.004469 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.004242    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1c3f5304-6120-4ec9-bc11-819124adc58c" (UID: "1c3f5304-6120-4ec9-bc11-819124adc58c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:27.004469 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.004335    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c3f5304-6120-4ec9-bc11-819124adc58c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1c3f5304-6120-4ec9-bc11-819124adc58c" (UID: "1c3f5304-6120-4ec9-bc11-819124adc58c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:57:27.004594 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.004489    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-data" (OuterVolumeSpecName: "istio-data") pod "1c3f5304-6120-4ec9-bc11-819124adc58c" (UID: "1c3f5304-6120-4ec9-bc11-819124adc58c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:27.004594 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.004563    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3f5304-6120-4ec9-bc11-819124adc58c-istiod-ca-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:27.004594 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.004582    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-workload-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:27.005515 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.004595    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-data\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:27.005515 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.004610    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-credential-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:27.005515 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.004623    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-workload-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:27.007116 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.007032    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1c3f5304-6120-4ec9-bc11-819124adc58c" (UID: "1c3f5304-6120-4ec9-bc11-819124adc58c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:27.007116 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.007066    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3f5304-6120-4ec9-bc11-819124adc58c-kube-api-access-zzdq7" (OuterVolumeSpecName: "kube-api-access-zzdq7") pod "1c3f5304-6120-4ec9-bc11-819124adc58c" (UID: "1c3f5304-6120-4ec9-bc11-819124adc58c"). InnerVolumeSpecName "kube-api-access-zzdq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:57:27.007293 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.007126    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1c3f5304-6120-4ec9-bc11-819124adc58c" (UID: "1c3f5304-6120-4ec9-bc11-819124adc58c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:57:27.007293 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.007132    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-token" (OuterVolumeSpecName: "istio-token") pod "1c3f5304-6120-4ec9-bc11-819124adc58c" (UID: "1c3f5304-6120-4ec9-bc11-819124adc58c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:57:27.105918 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.105880    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-envoy\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:27.105918 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.105915    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-podinfo\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:27.106123 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.105930    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzdq7\" (UniqueName: \"kubernetes.io/projected/1c3f5304-6120-4ec9-bc11-819124adc58c-kube-api-access-zzdq7\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:27.106123 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.105944    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3f5304-6120-4ec9-bc11-819124adc58c-istio-token\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:27.592624 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.592594    2564 generic.go:356] "Generic (PLEG): container finished" podID="1c3f5304-6120-4ec9-bc11-819124adc58c" containerID="0cd3533943ac8768459a11d60b74fbd6c34a03a0eb5cc01345ca3399595c8a62" exitCode=0
Sep 10 15:57:27.592849 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.592650    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"
Sep 10 15:57:27.592849 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.592687    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv" event={"ID":"1c3f5304-6120-4ec9-bc11-819124adc58c","Type":"ContainerDied","Data":"0cd3533943ac8768459a11d60b74fbd6c34a03a0eb5cc01345ca3399595c8a62"}
Sep 10 15:57:27.592849 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.592741    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv" event={"ID":"1c3f5304-6120-4ec9-bc11-819124adc58c","Type":"ContainerDied","Data":"8035ab56753683895cb28587e96547ed98e10e00848ea6a5fe632c23bbe72a7d"}
Sep 10 15:57:27.592849 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.592764    2564 scope.go:117] "RemoveContainer" containerID="0cd3533943ac8768459a11d60b74fbd6c34a03a0eb5cc01345ca3399595c8a62"
Sep 10 15:57:27.605380 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.605325    2564 scope.go:117] "RemoveContainer" containerID="47abb231c4baa45ff84f9263f555d538a01a0bf9a296d449294e89678fcec8e4"
Sep 10 15:57:27.622551 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.622521    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"]
Sep 10 15:57:27.623397 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.623268    2564 scope.go:117] "RemoveContainer" containerID="0cd3533943ac8768459a11d60b74fbd6c34a03a0eb5cc01345ca3399595c8a62"
Sep 10 15:57:27.625493 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:57:27.625462    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cd3533943ac8768459a11d60b74fbd6c34a03a0eb5cc01345ca3399595c8a62\": container with ID starting with 0cd3533943ac8768459a11d60b74fbd6c34a03a0eb5cc01345ca3399595c8a62 not found: ID does not exist" containerID="0cd3533943ac8768459a11d60b74fbd6c34a03a0eb5cc01345ca3399595c8a62"
Sep 10 15:57:27.625652 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.625620    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd3533943ac8768459a11d60b74fbd6c34a03a0eb5cc01345ca3399595c8a62"} err="failed to get container status \"0cd3533943ac8768459a11d60b74fbd6c34a03a0eb5cc01345ca3399595c8a62\": rpc error: code = NotFound desc = could not find container \"0cd3533943ac8768459a11d60b74fbd6c34a03a0eb5cc01345ca3399595c8a62\": container with ID starting with 0cd3533943ac8768459a11d60b74fbd6c34a03a0eb5cc01345ca3399595c8a62 not found: ID does not exist"
Sep 10 15:57:27.625787 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.625763    2564 scope.go:117] "RemoveContainer" containerID="47abb231c4baa45ff84f9263f555d538a01a0bf9a296d449294e89678fcec8e4"
Sep 10 15:57:27.626223 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:57:27.626190    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47abb231c4baa45ff84f9263f555d538a01a0bf9a296d449294e89678fcec8e4\": container with ID starting with 47abb231c4baa45ff84f9263f555d538a01a0bf9a296d449294e89678fcec8e4 not found: ID does not exist" containerID="47abb231c4baa45ff84f9263f555d538a01a0bf9a296d449294e89678fcec8e4"
Sep 10 15:57:27.626321 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.626223    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47abb231c4baa45ff84f9263f555d538a01a0bf9a296d449294e89678fcec8e4"} err="failed to get container status \"47abb231c4baa45ff84f9263f555d538a01a0bf9a296d449294e89678fcec8e4\": rpc error: code = NotFound desc = could not find container \"47abb231c4baa45ff84f9263f555d538a01a0bf9a296d449294e89678fcec8e4\": container with ID starting with 47abb231c4baa45ff84f9263f555d538a01a0bf9a296d449294e89678fcec8e4 not found: ID does not exist"
Sep 10 15:57:27.627646 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.627622    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-8d528235-openshift-default-85b468f5cb-mj9gv"]
Sep 10 15:57:27.673146 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:27.673115    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c3f5304-6120-4ec9-bc11-819124adc58c" path="/var/lib/kubelet/pods/1c3f5304-6120-4ec9-bc11-819124adc58c/volumes"
Sep 10 15:57:28.694590 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:28.694544    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5" podUID="01913550-0fd3-404e-bc81-8fab500c4f62" containerName="main" containerID="cri-o://e3c4b6617e9c9f9bb54fbb8eb3704d7fd4ecc2855dc4368661358c2b23bb8b2d" gracePeriod=45
Sep 10 15:57:28.949971 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:28.949916    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:57:29.025387 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.025355    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01913550-0fd3-404e-bc81-8fab500c4f62-tls-certs\") pod \"01913550-0fd3-404e-bc81-8fab500c4f62\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") "
Sep 10 15:57:29.025552 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.025429    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-home\") pod \"01913550-0fd3-404e-bc81-8fab500c4f62\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") "
Sep 10 15:57:29.025552 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.025463    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwptr\" (UniqueName: \"kubernetes.io/projected/01913550-0fd3-404e-bc81-8fab500c4f62-kube-api-access-jwptr\") pod \"01913550-0fd3-404e-bc81-8fab500c4f62\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") "
Sep 10 15:57:29.025552 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.025487    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-dshm\") pod \"01913550-0fd3-404e-bc81-8fab500c4f62\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") "
Sep 10 15:57:29.025552 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.025531    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-model-cache\") pod \"01913550-0fd3-404e-bc81-8fab500c4f62\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") "
Sep 10 15:57:29.025781 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.025617    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-tmp-dir\") pod \"01913550-0fd3-404e-bc81-8fab500c4f62\" (UID: \"01913550-0fd3-404e-bc81-8fab500c4f62\") "
Sep 10 15:57:29.025781 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.025758    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-home" (OuterVolumeSpecName: "home") pod "01913550-0fd3-404e-bc81-8fab500c4f62" (UID: "01913550-0fd3-404e-bc81-8fab500c4f62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:29.025888 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.025808    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-model-cache" (OuterVolumeSpecName: "model-cache") pod "01913550-0fd3-404e-bc81-8fab500c4f62" (UID: "01913550-0fd3-404e-bc81-8fab500c4f62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:29.025933 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.025888    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "01913550-0fd3-404e-bc81-8fab500c4f62" (UID: "01913550-0fd3-404e-bc81-8fab500c4f62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:29.026023 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.026006    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-model-cache\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:29.026064 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.026031    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-tmp-dir\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:29.026064 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.026044    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-home\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:29.027765 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.027741    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-dshm" (OuterVolumeSpecName: "dshm") pod "01913550-0fd3-404e-bc81-8fab500c4f62" (UID: "01913550-0fd3-404e-bc81-8fab500c4f62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:29.027857 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.027814    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01913550-0fd3-404e-bc81-8fab500c4f62-kube-api-access-jwptr" (OuterVolumeSpecName: "kube-api-access-jwptr") pod "01913550-0fd3-404e-bc81-8fab500c4f62" (UID: "01913550-0fd3-404e-bc81-8fab500c4f62"). InnerVolumeSpecName "kube-api-access-jwptr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:57:29.027857 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.027832    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01913550-0fd3-404e-bc81-8fab500c4f62-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01913550-0fd3-404e-bc81-8fab500c4f62" (UID: "01913550-0fd3-404e-bc81-8fab500c4f62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:57:29.126747 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.126717    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01913550-0fd3-404e-bc81-8fab500c4f62-tls-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:29.126747 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.126745    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwptr\" (UniqueName: \"kubernetes.io/projected/01913550-0fd3-404e-bc81-8fab500c4f62-kube-api-access-jwptr\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:29.126747 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.126756    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01913550-0fd3-404e-bc81-8fab500c4f62-dshm\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:29.611756 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.611713    2564 generic.go:356] "Generic (PLEG): container finished" podID="01913550-0fd3-404e-bc81-8fab500c4f62" containerID="e3c4b6617e9c9f9bb54fbb8eb3704d7fd4ecc2855dc4368661358c2b23bb8b2d" exitCode=0
Sep 10 15:57:29.611945 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.611762    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5" event={"ID":"01913550-0fd3-404e-bc81-8fab500c4f62","Type":"ContainerDied","Data":"e3c4b6617e9c9f9bb54fbb8eb3704d7fd4ecc2855dc4368661358c2b23bb8b2d"}
Sep 10 15:57:29.611945 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.611770    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"
Sep 10 15:57:29.611945 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.611809    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5" event={"ID":"01913550-0fd3-404e-bc81-8fab500c4f62","Type":"ContainerDied","Data":"68332130d6398939ae91a87373d481c9e79c5af684924d28a9d5449a5275a2bc"}
Sep 10 15:57:29.611945 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.611829    2564 scope.go:117] "RemoveContainer" containerID="e3c4b6617e9c9f9bb54fbb8eb3704d7fd4ecc2855dc4368661358c2b23bb8b2d"
Sep 10 15:57:29.622393 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.622370    2564 scope.go:117] "RemoveContainer" containerID="e3c4b6617e9c9f9bb54fbb8eb3704d7fd4ecc2855dc4368661358c2b23bb8b2d"
Sep 10 15:57:29.622749 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:57:29.622728    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c4b6617e9c9f9bb54fbb8eb3704d7fd4ecc2855dc4368661358c2b23bb8b2d\": container with ID starting with e3c4b6617e9c9f9bb54fbb8eb3704d7fd4ecc2855dc4368661358c2b23bb8b2d not found: ID does not exist" containerID="e3c4b6617e9c9f9bb54fbb8eb3704d7fd4ecc2855dc4368661358c2b23bb8b2d"
Sep 10 15:57:29.622821 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.622758    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c4b6617e9c9f9bb54fbb8eb3704d7fd4ecc2855dc4368661358c2b23bb8b2d"} err="failed to get container status \"e3c4b6617e9c9f9bb54fbb8eb3704d7fd4ecc2855dc4368661358c2b23bb8b2d\": rpc error: code = NotFound desc = could not find container \"e3c4b6617e9c9f9bb54fbb8eb3704d7fd4ecc2855dc4368661358c2b23bb8b2d\": container with ID starting with e3c4b6617e9c9f9bb54fbb8eb3704d7fd4ecc2855dc4368661358c2b23bb8b2d not found: ID does not exist"
Sep 10 15:57:29.637476 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.637443    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"]
Sep 10 15:57:29.641746 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.641720    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-87a024/autores-model-87a024-kserve-6bc5f968c5-m7wj5"]
Sep 10 15:57:29.673067 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:29.673026    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01913550-0fd3-404e-bc81-8fab500c4f62" path="/var/lib/kubelet/pods/01913550-0fd3-404e-bc81-8fab500c4f62/volumes"
Sep 10 15:57:33.001684 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.001641    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"]
Sep 10 15:57:33.002122 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.001925    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs" podUID="e5116818-0a78-4fe9-8b84-b587b1a3d080" containerName="istio-proxy" containerID="cri-o://330ae4ba68b3a6f8c2ce7432a42189f9e63116fa6c42c611f65044fd65a506ff" gracePeriod=30
Sep 10 15:57:33.758528 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.758493    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"]
Sep 10 15:57:33.759336 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.759308    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01913550-0fd3-404e-bc81-8fab500c4f62" containerName="main"
Sep 10 15:57:33.759539 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.759518    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="01913550-0fd3-404e-bc81-8fab500c4f62" containerName="main"
Sep 10 15:57:33.759655 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.759550    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fc405ea-b5f4-4c99-ba4c-bc98aac499ea" containerName="payload-processing"
Sep 10 15:57:33.759655 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.759558    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fc405ea-b5f4-4c99-ba4c-bc98aac499ea" containerName="payload-processing"
Sep 10 15:57:33.759655 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.759582    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c3f5304-6120-4ec9-bc11-819124adc58c" containerName="istio-proxy"
Sep 10 15:57:33.759655 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.759587    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c3f5304-6120-4ec9-bc11-819124adc58c" containerName="istio-proxy"
Sep 10 15:57:33.759655 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.759595    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55016ec4-918c-4f80-bd6b-e8f2b8a99b01" containerName="revoke-keys"
Sep 10 15:57:33.759655 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.759602    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="55016ec4-918c-4f80-bd6b-e8f2b8a99b01" containerName="revoke-keys"
Sep 10 15:57:33.759655 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.759623    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c3f5304-6120-4ec9-bc11-819124adc58c" containerName="istio-proxy"
Sep 10 15:57:33.759655 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.759627    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c3f5304-6120-4ec9-bc11-819124adc58c" containerName="istio-proxy"
Sep 10 15:57:33.759939 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.759739    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c3f5304-6120-4ec9-bc11-819124adc58c" containerName="istio-proxy"
Sep 10 15:57:33.759939 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.759750    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="01913550-0fd3-404e-bc81-8fab500c4f62" containerName="main"
Sep 10 15:57:33.759939 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.759755    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c3f5304-6120-4ec9-bc11-819124adc58c" containerName="istio-proxy"
Sep 10 15:57:33.759939 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.759760    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="55016ec4-918c-4f80-bd6b-e8f2b8a99b01" containerName="revoke-keys"
Sep 10 15:57:33.759939 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.759766    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fc405ea-b5f4-4c99-ba4c-bc98aac499ea" containerName="payload-processing"
Sep 10 15:57:33.764592 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.764567    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.767268 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.767244    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-61fba2-openshift-default-dockercfg-pmh2n\""
Sep 10 15:57:33.775210 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.775187    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"]
Sep 10 15:57:33.872743 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.872695    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq4rk\" (UniqueName: \"kubernetes.io/projected/97ed3c88-bfeb-4286-a429-e0d406cf6931-kube-api-access-mq4rk\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.872743 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.872751    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-credential-socket\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.872977 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.872810    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-data\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.872977 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.872851    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-token\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.872977 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.872877    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ed3c88-bfeb-4286-a429-e0d406cf6931-istiod-ca-cert\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.872977 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.872898    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-envoy\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.872977 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.872959    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-workload-certs\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.873246 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.872981    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-workload-socket\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.873246 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.873084    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-podinfo\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.973986 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.973947    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-token\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.973986 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.973991    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ed3c88-bfeb-4286-a429-e0d406cf6931-istiod-ca-cert\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.974268 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.974012    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-envoy\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.974268 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.974050    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-workload-certs\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.974268 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.974075    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-workload-socket\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.974268 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.974112    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-podinfo\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.974268 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.974205    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq4rk\" (UniqueName: \"kubernetes.io/projected/97ed3c88-bfeb-4286-a429-e0d406cf6931-kube-api-access-mq4rk\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.974268 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.974235    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-credential-socket\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.974602 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.974270    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-data\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.974602 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.974546    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-workload-socket\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.974602 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.974579    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-workload-certs\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.974809 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.974791    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-data\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.974877 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.974804    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ed3c88-bfeb-4286-a429-e0d406cf6931-istiod-ca-cert\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.974877 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.974830    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-credential-socket\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.976849 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.976826    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-envoy\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.976976 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.976957    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-podinfo\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.983544 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.983517    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq4rk\" (UniqueName: \"kubernetes.io/projected/97ed3c88-bfeb-4286-a429-e0d406cf6931-kube-api-access-mq4rk\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:33.983810 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:33.983792    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-token\") pod \"e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") " pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:34.077413 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:34.077315    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:34.366486 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:34.366464    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"]
Sep 10 15:57:34.368399 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:57:34.368372    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ed3c88_bfeb_4286_a429_e0d406cf6931.slice/crio-3ec81948b4bbe083e80e1b83f8c084ba47ac5a9f1983ae6768a4cda7109e647c WatchSource:0}: Error finding container 3ec81948b4bbe083e80e1b83f8c084ba47ac5a9f1983ae6768a4cda7109e647c: Status 404 returned error can't find the container with id 3ec81948b4bbe083e80e1b83f8c084ba47ac5a9f1983ae6768a4cda7109e647c
Sep 10 15:57:34.371004 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:34.370969    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:57:34.371100 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:34.371036    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:57:34.371100 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:34.371064    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:57:34.644479 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:34.644376    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd" event={"ID":"97ed3c88-bfeb-4286-a429-e0d406cf6931","Type":"ContainerStarted","Data":"36ea4b9daf5fa88b53e737d1f27c668a51e292f35d3bd2c4f30a9d074b5c6606"}
Sep 10 15:57:34.644479 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:34.644446    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd" event={"ID":"97ed3c88-bfeb-4286-a429-e0d406cf6931","Type":"ContainerStarted","Data":"3ec81948b4bbe083e80e1b83f8c084ba47ac5a9f1983ae6768a4cda7109e647c"}
Sep 10 15:57:34.671890 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:34.671828    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd" podStartSLOduration=1.671805918 podStartE2EDuration="1.671805918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:57:34.669066838 +0000 UTC m=+933.473386102" watchObservedRunningTime="2026-09-10 15:57:34.671805918 +0000 UTC m=+933.476125180"
Sep 10 15:57:35.077939 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:35.077907    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:35.083285 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:35.083260    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:35.651129 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:35.651099    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:35.652186 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:35.652141    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:35.675590 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:35.675563    2564 patch_prober.go:28] interesting pod/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 15:57:35.675715 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:35.675610    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs" podUID="e5116818-0a78-4fe9-8b84-b587b1a3d080" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 15:57:38.270656 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.270588    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:57:38.422441 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.422404    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-workload-certs\") pod \"e5116818-0a78-4fe9-8b84-b587b1a3d080\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") "
Sep 10 15:57:38.422441 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.422444    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-credential-socket\") pod \"e5116818-0a78-4fe9-8b84-b587b1a3d080\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") "
Sep 10 15:57:38.422693 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.422476    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5116818-0a78-4fe9-8b84-b587b1a3d080-istiod-ca-cert\") pod \"e5116818-0a78-4fe9-8b84-b587b1a3d080\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") "
Sep 10 15:57:38.422693 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.422536    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-data\") pod \"e5116818-0a78-4fe9-8b84-b587b1a3d080\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") "
Sep 10 15:57:38.422693 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.422589    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-envoy\") pod \"e5116818-0a78-4fe9-8b84-b587b1a3d080\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") "
Sep 10 15:57:38.422693 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.422664    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-podinfo\") pod \"e5116818-0a78-4fe9-8b84-b587b1a3d080\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") "
Sep 10 15:57:38.422693 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.422690    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-workload-socket\") pod \"e5116818-0a78-4fe9-8b84-b587b1a3d080\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") "
Sep 10 15:57:38.422942 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.422718    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-token\") pod \"e5116818-0a78-4fe9-8b84-b587b1a3d080\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") "
Sep 10 15:57:38.422942 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.422743    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8qdt\" (UniqueName: \"kubernetes.io/projected/e5116818-0a78-4fe9-8b84-b587b1a3d080-kube-api-access-t8qdt\") pod \"e5116818-0a78-4fe9-8b84-b587b1a3d080\" (UID: \"e5116818-0a78-4fe9-8b84-b587b1a3d080\") "
Sep 10 15:57:38.422942 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.422742    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5116818-0a78-4fe9-8b84-b587b1a3d080" (UID: "e5116818-0a78-4fe9-8b84-b587b1a3d080"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:38.422942 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.422895    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5116818-0a78-4fe9-8b84-b587b1a3d080" (UID: "e5116818-0a78-4fe9-8b84-b587b1a3d080"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:38.423152 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.422993    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5116818-0a78-4fe9-8b84-b587b1a3d080-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5116818-0a78-4fe9-8b84-b587b1a3d080" (UID: "e5116818-0a78-4fe9-8b84-b587b1a3d080"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:57:38.423152 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.423089    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5116818-0a78-4fe9-8b84-b587b1a3d080" (UID: "e5116818-0a78-4fe9-8b84-b587b1a3d080"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:38.423282 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.423252    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5116818-0a78-4fe9-8b84-b587b1a3d080" (UID: "e5116818-0a78-4fe9-8b84-b587b1a3d080"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:38.423449 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.423364    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-workload-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:38.423560 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.423528    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-credential-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:38.423613 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.423564    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5116818-0a78-4fe9-8b84-b587b1a3d080-istiod-ca-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:38.423613 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.423578    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-data\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:38.423613 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.423590    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-workload-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:38.425050 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.424986    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5116818-0a78-4fe9-8b84-b587b1a3d080" (UID: "e5116818-0a78-4fe9-8b84-b587b1a3d080"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:38.425255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.425061    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5116818-0a78-4fe9-8b84-b587b1a3d080-kube-api-access-t8qdt" (OuterVolumeSpecName: "kube-api-access-t8qdt") pod "e5116818-0a78-4fe9-8b84-b587b1a3d080" (UID: "e5116818-0a78-4fe9-8b84-b587b1a3d080"). InnerVolumeSpecName "kube-api-access-t8qdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:57:38.425815 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.425777    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5116818-0a78-4fe9-8b84-b587b1a3d080" (UID: "e5116818-0a78-4fe9-8b84-b587b1a3d080"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:57:38.425930 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.425868    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5116818-0a78-4fe9-8b84-b587b1a3d080" (UID: "e5116818-0a78-4fe9-8b84-b587b1a3d080"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:57:38.524237 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.524144    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-envoy\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:38.524237 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.524190    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-podinfo\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:38.524237 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.524201    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5116818-0a78-4fe9-8b84-b587b1a3d080-istio-token\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:38.524237 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.524210    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8qdt\" (UniqueName: \"kubernetes.io/projected/e5116818-0a78-4fe9-8b84-b587b1a3d080-kube-api-access-t8qdt\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:57:38.671187 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.671137    2564 generic.go:356] "Generic (PLEG): container finished" podID="e5116818-0a78-4fe9-8b84-b587b1a3d080" containerID="330ae4ba68b3a6f8c2ce7432a42189f9e63116fa6c42c611f65044fd65a506ff" exitCode=0
Sep 10 15:57:38.671365 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.671198    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"
Sep 10 15:57:38.671365 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.671201    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs" event={"ID":"e5116818-0a78-4fe9-8b84-b587b1a3d080","Type":"ContainerDied","Data":"330ae4ba68b3a6f8c2ce7432a42189f9e63116fa6c42c611f65044fd65a506ff"}
Sep 10 15:57:38.671365 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.671254    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs" event={"ID":"e5116818-0a78-4fe9-8b84-b587b1a3d080","Type":"ContainerDied","Data":"97aeec8a290bf4e999e2fe29b1a2a415a1451d7d134e037aae18b658127f8a5f"}
Sep 10 15:57:38.671365 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.671285    2564 scope.go:117] "RemoveContainer" containerID="330ae4ba68b3a6f8c2ce7432a42189f9e63116fa6c42c611f65044fd65a506ff"
Sep 10 15:57:38.681710 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.681693    2564 scope.go:117] "RemoveContainer" containerID="a2a768687934982ce613cb10c742937c23a62035408ab610c1acfa3bf9152d37"
Sep 10 15:57:38.691030 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.691008    2564 scope.go:117] "RemoveContainer" containerID="330ae4ba68b3a6f8c2ce7432a42189f9e63116fa6c42c611f65044fd65a506ff"
Sep 10 15:57:38.691406 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:57:38.691386    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330ae4ba68b3a6f8c2ce7432a42189f9e63116fa6c42c611f65044fd65a506ff\": container with ID starting with 330ae4ba68b3a6f8c2ce7432a42189f9e63116fa6c42c611f65044fd65a506ff not found: ID does not exist" containerID="330ae4ba68b3a6f8c2ce7432a42189f9e63116fa6c42c611f65044fd65a506ff"
Sep 10 15:57:38.691497 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.691420    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330ae4ba68b3a6f8c2ce7432a42189f9e63116fa6c42c611f65044fd65a506ff"} err="failed to get container status \"330ae4ba68b3a6f8c2ce7432a42189f9e63116fa6c42c611f65044fd65a506ff\": rpc error: code = NotFound desc = could not find container \"330ae4ba68b3a6f8c2ce7432a42189f9e63116fa6c42c611f65044fd65a506ff\": container with ID starting with 330ae4ba68b3a6f8c2ce7432a42189f9e63116fa6c42c611f65044fd65a506ff not found: ID does not exist"
Sep 10 15:57:38.691497 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.691445    2564 scope.go:117] "RemoveContainer" containerID="a2a768687934982ce613cb10c742937c23a62035408ab610c1acfa3bf9152d37"
Sep 10 15:57:38.691720 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:57:38.691703    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2a768687934982ce613cb10c742937c23a62035408ab610c1acfa3bf9152d37\": container with ID starting with a2a768687934982ce613cb10c742937c23a62035408ab610c1acfa3bf9152d37 not found: ID does not exist" containerID="a2a768687934982ce613cb10c742937c23a62035408ab610c1acfa3bf9152d37"
Sep 10 15:57:38.691760 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.691727    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2a768687934982ce613cb10c742937c23a62035408ab610c1acfa3bf9152d37"} err="failed to get container status \"a2a768687934982ce613cb10c742937c23a62035408ab610c1acfa3bf9152d37\": rpc error: code = NotFound desc = could not find container \"a2a768687934982ce613cb10c742937c23a62035408ab610c1acfa3bf9152d37\": container with ID starting with a2a768687934982ce613cb10c742937c23a62035408ab610c1acfa3bf9152d37 not found: ID does not exist"
Sep 10 15:57:38.701211 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.701179    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"]
Sep 10 15:57:38.707116 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:38.707092    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-87a024-openshift-default-8554f8f878-j4fjs"]
Sep 10 15:57:39.674208 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:39.674153    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5116818-0a78-4fe9-8b84-b587b1a3d080" path="/var/lib/kubelet/pods/e5116818-0a78-4fe9-8b84-b587b1a3d080/volumes"
Sep 10 15:57:51.692364 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:51.692322    2564 patch_prober.go:28] interesting pod/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 15:57:51.692784 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:51.692405    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd" podUID="97ed3c88-bfeb-4286-a429-e0d406cf6931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 15:57:52.765838 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:52.765797    2564 generic.go:356] "Generic (PLEG): container finished" podID="97ed3c88-bfeb-4286-a429-e0d406cf6931" containerID="36ea4b9daf5fa88b53e737d1f27c668a51e292f35d3bd2c4f30a9d074b5c6606" exitCode=0
Sep 10 15:57:52.766324 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:52.765860    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd" event={"ID":"97ed3c88-bfeb-4286-a429-e0d406cf6931","Type":"ContainerDied","Data":"36ea4b9daf5fa88b53e737d1f27c668a51e292f35d3bd2c4f30a9d074b5c6606"}
Sep 10 15:57:52.766324 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:52.766267    2564 scope.go:117] "RemoveContainer" containerID="36ea4b9daf5fa88b53e737d1f27c668a51e292f35d3bd2c4f30a9d074b5c6606"
Sep 10 15:57:52.767476 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:52.767445    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:57:52.767549 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:52.767518    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:57:52.767589 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:52.767549    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:57:53.774377 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:53.774342    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd" event={"ID":"97ed3c88-bfeb-4286-a429-e0d406cf6931","Type":"ContainerStarted","Data":"13a18acf2a74116a482f9d9c167c25d568e573b01f7f2f364cfebbf8707b079d"}
Sep 10 15:57:54.078063 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:54.077963    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:54.078270 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:54.078152    2564 patch_prober.go:28] interesting pod/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 10 15:57:54.078270 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:54.078237    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd" podUID="97ed3c88-bfeb-4286-a429-e0d406cf6931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 10 15:57:55.077728 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:55.077689    2564 patch_prober.go:28] interesting pod/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 10 15:57:55.078142 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:55.077755    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd" podUID="97ed3c88-bfeb-4286-a429-e0d406cf6931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 10 15:57:56.078118 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:56.078076    2564 patch_prober.go:28] interesting pod/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 10 15:57:56.078552 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:56.078150    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd" podUID="97ed3c88-bfeb-4286-a429-e0d406cf6931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 10 15:57:57.077714 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:57.077678    2564 patch_prober.go:28] interesting pod/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 10 15:57:57.077936 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:57.077738    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd" podUID="97ed3c88-bfeb-4286-a429-e0d406cf6931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 10 15:57:58.078144 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:58.078101    2564 patch_prober.go:28] interesting pod/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 10 15:57:58.078582 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:58.078207    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd" podUID="97ed3c88-bfeb-4286-a429-e0d406cf6931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 10 15:57:59.082328 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:59.082297    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:59.082727 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:59.082655    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:57:59.083509 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:57:59.083493    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:58:34.128354 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:34.128275    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-61fba2-bb76j"]
Sep 10 15:58:34.128977 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:34.128960    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5116818-0a78-4fe9-8b84-b587b1a3d080" containerName="istio-proxy"
Sep 10 15:58:34.129024 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:34.128981    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5116818-0a78-4fe9-8b84-b587b1a3d080" containerName="istio-proxy"
Sep 10 15:58:34.129114 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:34.129102    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5116818-0a78-4fe9-8b84-b587b1a3d080" containerName="istio-proxy"
Sep 10 15:58:34.129152 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:34.129119    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5116818-0a78-4fe9-8b84-b587b1a3d080" containerName="istio-proxy"
Sep 10 15:58:34.132548 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:34.132529    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-61fba2-bb76j"
Sep 10 15:58:34.134997 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:34.134979    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2wghg\""
Sep 10 15:58:34.140182 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:34.140144    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-61fba2-bb76j"]
Sep 10 15:58:34.200291 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:34.200256    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d7992e-3ace-46f1-8763-051c9f4bc758-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-61fba2-bb76j\" (UID: \"f7d7992e-3ace-46f1-8763-051c9f4bc758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-61fba2-bb76j"
Sep 10 15:58:34.301074 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:34.301037    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d7992e-3ace-46f1-8763-051c9f4bc758-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-61fba2-bb76j\" (UID: \"f7d7992e-3ace-46f1-8763-051c9f4bc758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-61fba2-bb76j"
Sep 10 15:58:34.301664 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:34.301646    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d7992e-3ace-46f1-8763-051c9f4bc758-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-61fba2-bb76j\" (UID: \"f7d7992e-3ace-46f1-8763-051c9f4bc758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-61fba2-bb76j"
Sep 10 15:58:34.446455 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:34.446422    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-61fba2-bb76j"
Sep 10 15:58:34.723239 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:34.723214    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-61fba2-bb76j"]
Sep 10 15:58:34.724985 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:58:34.724952    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d7992e_3ace_46f1_8763_051c9f4bc758.slice/crio-ac098a7d11b5ce1576f2123d48115d231d4f6399338fe7ac93c4072c9b1f5a94 WatchSource:0}: Error finding container ac098a7d11b5ce1576f2123d48115d231d4f6399338fe7ac93c4072c9b1f5a94: Status 404 returned error can't find the container with id ac098a7d11b5ce1576f2123d48115d231d4f6399338fe7ac93c4072c9b1f5a94
Sep 10 15:58:35.028385 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:35.028353    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-61fba2-bb76j" event={"ID":"f7d7992e-3ace-46f1-8763-051c9f4bc758","Type":"ContainerStarted","Data":"0ae9f07be9f36b41cda04cd292d613c40819e585eb2b0a334dd9483729100d9d"}
Sep 10 15:58:35.028511 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:35.028433    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-61fba2-bb76j" event={"ID":"f7d7992e-3ace-46f1-8763-051c9f4bc758","Type":"ContainerStarted","Data":"ac098a7d11b5ce1576f2123d48115d231d4f6399338fe7ac93c4072c9b1f5a94"}
Sep 10 15:58:35.047290 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:35.047242    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-61fba2-bb76j" podStartSLOduration=1.047227214 podStartE2EDuration="1.047227214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:58:35.044964805 +0000 UTC m=+993.849284066" watchObservedRunningTime="2026-09-10 15:58:35.047227214 +0000 UTC m=+993.851546477"
Sep 10 15:58:36.035240 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:36.035207    2564 generic.go:356] "Generic (PLEG): container finished" podID="f7d7992e-3ace-46f1-8763-051c9f4bc758" containerID="0ae9f07be9f36b41cda04cd292d613c40819e585eb2b0a334dd9483729100d9d" exitCode=0
Sep 10 15:58:36.035634 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:36.035246    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-61fba2-bb76j" event={"ID":"f7d7992e-3ace-46f1-8763-051c9f4bc758","Type":"ContainerDied","Data":"0ae9f07be9f36b41cda04cd292d613c40819e585eb2b0a334dd9483729100d9d"}
Sep 10 15:58:37.184858 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:37.184834    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-61fba2-bb76j"
Sep 10 15:58:37.235981 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:37.235936    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d7992e-3ace-46f1-8763-051c9f4bc758-maas-api-service-ca\") pod \"f7d7992e-3ace-46f1-8763-051c9f4bc758\" (UID: \"f7d7992e-3ace-46f1-8763-051c9f4bc758\") "
Sep 10 15:58:37.236385 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:37.236355    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d7992e-3ace-46f1-8763-051c9f4bc758-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7d7992e-3ace-46f1-8763-051c9f4bc758" (UID: "f7d7992e-3ace-46f1-8763-051c9f4bc758"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:58:37.336988 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:37.336889    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d7992e-3ace-46f1-8763-051c9f4bc758-maas-api-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:58:38.048350 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:38.048268    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-61fba2-bb76j"
Sep 10 15:58:38.048500 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:38.048267    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-61fba2-bb76j" event={"ID":"f7d7992e-3ace-46f1-8763-051c9f4bc758","Type":"ContainerDied","Data":"ac098a7d11b5ce1576f2123d48115d231d4f6399338fe7ac93c4072c9b1f5a94"}
Sep 10 15:58:38.048500 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:38.048391    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac098a7d11b5ce1576f2123d48115d231d4f6399338fe7ac93c4072c9b1f5a94"
Sep 10 15:58:44.907077 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:44.907047    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-61fba2-bb76j"]
Sep 10 15:58:44.910873 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:44.910849    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-61fba2-bb76j"]
Sep 10 15:58:45.673875 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:45.673843    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7d7992e-3ace-46f1-8763-051c9f4bc758" path="/var/lib/kubelet/pods/f7d7992e-3ace-46f1-8763-051c9f4bc758/volumes"
Sep 10 15:58:52.359946 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:52.359913    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"]
Sep 10 15:58:52.360412 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:52.360125    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd" podUID="97ed3c88-bfeb-4286-a429-e0d406cf6931" containerName="istio-proxy" containerID="cri-o://13a18acf2a74116a482f9d9c167c25d568e573b01f7f2f364cfebbf8707b079d" gracePeriod=30
Sep 10 15:58:57.631632 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.631567    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:58:57.755531 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.755498    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ed3c88-bfeb-4286-a429-e0d406cf6931-istiod-ca-cert\") pod \"97ed3c88-bfeb-4286-a429-e0d406cf6931\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") "
Sep 10 15:58:57.755703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.755540    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-token\") pod \"97ed3c88-bfeb-4286-a429-e0d406cf6931\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") "
Sep 10 15:58:57.755703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.755565    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-data\") pod \"97ed3c88-bfeb-4286-a429-e0d406cf6931\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") "
Sep 10 15:58:57.755703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.755590    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-podinfo\") pod \"97ed3c88-bfeb-4286-a429-e0d406cf6931\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") "
Sep 10 15:58:57.755703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.755640    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq4rk\" (UniqueName: \"kubernetes.io/projected/97ed3c88-bfeb-4286-a429-e0d406cf6931-kube-api-access-mq4rk\") pod \"97ed3c88-bfeb-4286-a429-e0d406cf6931\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") "
Sep 10 15:58:57.755703 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.755683    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-credential-socket\") pod \"97ed3c88-bfeb-4286-a429-e0d406cf6931\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") "
Sep 10 15:58:57.756011 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.755738    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-envoy\") pod \"97ed3c88-bfeb-4286-a429-e0d406cf6931\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") "
Sep 10 15:58:57.756011 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.755760    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-workload-socket\") pod \"97ed3c88-bfeb-4286-a429-e0d406cf6931\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") "
Sep 10 15:58:57.756011 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.755877    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-data" (OuterVolumeSpecName: "istio-data") pod "97ed3c88-bfeb-4286-a429-e0d406cf6931" (UID: "97ed3c88-bfeb-4286-a429-e0d406cf6931"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:58:57.756011 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.755926    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ed3c88-bfeb-4286-a429-e0d406cf6931-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "97ed3c88-bfeb-4286-a429-e0d406cf6931" (UID: "97ed3c88-bfeb-4286-a429-e0d406cf6931"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:58:57.756271 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.756086    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "97ed3c88-bfeb-4286-a429-e0d406cf6931" (UID: "97ed3c88-bfeb-4286-a429-e0d406cf6931"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:58:57.756271 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.756104    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "97ed3c88-bfeb-4286-a429-e0d406cf6931" (UID: "97ed3c88-bfeb-4286-a429-e0d406cf6931"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:58:57.756271 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.756127    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-workload-certs\") pod \"97ed3c88-bfeb-4286-a429-e0d406cf6931\" (UID: \"97ed3c88-bfeb-4286-a429-e0d406cf6931\") "
Sep 10 15:58:57.756595 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.756303    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "97ed3c88-bfeb-4286-a429-e0d406cf6931" (UID: "97ed3c88-bfeb-4286-a429-e0d406cf6931"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:58:57.756747 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.756726    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-credential-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:58:57.756814 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.756753    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-workload-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:58:57.756814 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.756769    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-workload-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:58:57.756814 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.756784    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ed3c88-bfeb-4286-a429-e0d406cf6931-istiod-ca-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:58:57.756814 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.756797    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-data\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:58:57.758086 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.758065    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "97ed3c88-bfeb-4286-a429-e0d406cf6931" (UID: "97ed3c88-bfeb-4286-a429-e0d406cf6931"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:58:57.758208 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.758149    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ed3c88-bfeb-4286-a429-e0d406cf6931-kube-api-access-mq4rk" (OuterVolumeSpecName: "kube-api-access-mq4rk") pod "97ed3c88-bfeb-4286-a429-e0d406cf6931" (UID: "97ed3c88-bfeb-4286-a429-e0d406cf6931"). InnerVolumeSpecName "kube-api-access-mq4rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:58:57.758403 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.758386    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "97ed3c88-bfeb-4286-a429-e0d406cf6931" (UID: "97ed3c88-bfeb-4286-a429-e0d406cf6931"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:58:57.758445 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.758421    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-token" (OuterVolumeSpecName: "istio-token") pod "97ed3c88-bfeb-4286-a429-e0d406cf6931" (UID: "97ed3c88-bfeb-4286-a429-e0d406cf6931"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:58:57.857949 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.857916    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq4rk\" (UniqueName: \"kubernetes.io/projected/97ed3c88-bfeb-4286-a429-e0d406cf6931-kube-api-access-mq4rk\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:58:57.857949 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.857944    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-envoy\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:58:57.857949 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.857954    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-token\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:58:57.858209 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:57.857962    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ed3c88-bfeb-4286-a429-e0d406cf6931-istio-podinfo\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:58:58.168643 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:58.168606    2564 generic.go:356] "Generic (PLEG): container finished" podID="97ed3c88-bfeb-4286-a429-e0d406cf6931" containerID="13a18acf2a74116a482f9d9c167c25d568e573b01f7f2f364cfebbf8707b079d" exitCode=0
Sep 10 15:58:58.168845 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:58.168668    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd" event={"ID":"97ed3c88-bfeb-4286-a429-e0d406cf6931","Type":"ContainerDied","Data":"13a18acf2a74116a482f9d9c167c25d568e573b01f7f2f364cfebbf8707b079d"}
Sep 10 15:58:58.168845 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:58.168722    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd" event={"ID":"97ed3c88-bfeb-4286-a429-e0d406cf6931","Type":"ContainerDied","Data":"3ec81948b4bbe083e80e1b83f8c084ba47ac5a9f1983ae6768a4cda7109e647c"}
Sep 10 15:58:58.168845 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:58.168736    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"
Sep 10 15:58:58.168845 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:58.168743    2564 scope.go:117] "RemoveContainer" containerID="13a18acf2a74116a482f9d9c167c25d568e573b01f7f2f364cfebbf8707b079d"
Sep 10 15:58:58.179605 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:58.179588    2564 scope.go:117] "RemoveContainer" containerID="36ea4b9daf5fa88b53e737d1f27c668a51e292f35d3bd2c4f30a9d074b5c6606"
Sep 10 15:58:58.188985 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:58.188967    2564 scope.go:117] "RemoveContainer" containerID="13a18acf2a74116a482f9d9c167c25d568e573b01f7f2f364cfebbf8707b079d"
Sep 10 15:58:58.189347 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:58:58.189327    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a18acf2a74116a482f9d9c167c25d568e573b01f7f2f364cfebbf8707b079d\": container with ID starting with 13a18acf2a74116a482f9d9c167c25d568e573b01f7f2f364cfebbf8707b079d not found: ID does not exist" containerID="13a18acf2a74116a482f9d9c167c25d568e573b01f7f2f364cfebbf8707b079d"
Sep 10 15:58:58.189476 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:58.189357    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a18acf2a74116a482f9d9c167c25d568e573b01f7f2f364cfebbf8707b079d"} err="failed to get container status \"13a18acf2a74116a482f9d9c167c25d568e573b01f7f2f364cfebbf8707b079d\": rpc error: code = NotFound desc = could not find container \"13a18acf2a74116a482f9d9c167c25d568e573b01f7f2f364cfebbf8707b079d\": container with ID starting with 13a18acf2a74116a482f9d9c167c25d568e573b01f7f2f364cfebbf8707b079d not found: ID does not exist"
Sep 10 15:58:58.189476 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:58.189374    2564 scope.go:117] "RemoveContainer" containerID="36ea4b9daf5fa88b53e737d1f27c668a51e292f35d3bd2c4f30a9d074b5c6606"
Sep 10 15:58:58.189723 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:58:58.189697    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36ea4b9daf5fa88b53e737d1f27c668a51e292f35d3bd2c4f30a9d074b5c6606\": container with ID starting with 36ea4b9daf5fa88b53e737d1f27c668a51e292f35d3bd2c4f30a9d074b5c6606 not found: ID does not exist" containerID="36ea4b9daf5fa88b53e737d1f27c668a51e292f35d3bd2c4f30a9d074b5c6606"
Sep 10 15:58:58.189772 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:58.189732    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36ea4b9daf5fa88b53e737d1f27c668a51e292f35d3bd2c4f30a9d074b5c6606"} err="failed to get container status \"36ea4b9daf5fa88b53e737d1f27c668a51e292f35d3bd2c4f30a9d074b5c6606\": rpc error: code = NotFound desc = could not find container \"36ea4b9daf5fa88b53e737d1f27c668a51e292f35d3bd2c4f30a9d074b5c6606\": container with ID starting with 36ea4b9daf5fa88b53e737d1f27c668a51e292f35d3bd2c4f30a9d074b5c6606 not found: ID does not exist"
Sep 10 15:58:58.206263 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:58.206237    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"]
Sep 10 15:58:58.210924 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:58.210899    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-61fba2-openshift-default-755d6f7d96-g42xd"]
Sep 10 15:58:59.673402 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:58:59.673365    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97ed3c88-bfeb-4286-a429-e0d406cf6931" path="/var/lib/kubelet/pods/97ed3c88-bfeb-4286-a429-e0d406cf6931/volumes"
Sep 10 15:59:03.359062 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.359026    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"]
Sep 10 15:59:03.359480 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.359466    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7d7992e-3ace-46f1-8763-051c9f4bc758" containerName="revoke-keys"
Sep 10 15:59:03.359522 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.359482    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7d7992e-3ace-46f1-8763-051c9f4bc758" containerName="revoke-keys"
Sep 10 15:59:03.359522 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.359496    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ed3c88-bfeb-4286-a429-e0d406cf6931" containerName="istio-proxy"
Sep 10 15:59:03.359522 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.359503    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ed3c88-bfeb-4286-a429-e0d406cf6931" containerName="istio-proxy"
Sep 10 15:59:03.359522 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.359519    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5116818-0a78-4fe9-8b84-b587b1a3d080" containerName="istio-proxy"
Sep 10 15:59:03.359647 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.359527    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5116818-0a78-4fe9-8b84-b587b1a3d080" containerName="istio-proxy"
Sep 10 15:59:03.359647 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.359590    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="97ed3c88-bfeb-4286-a429-e0d406cf6931" containerName="istio-proxy"
Sep 10 15:59:03.359647 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.359597    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7d7992e-3ace-46f1-8763-051c9f4bc758" containerName="revoke-keys"
Sep 10 15:59:03.364440 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.364416    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.367240 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.367215    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9a70cb1a-openshift-default-dockercfg-sk2bj\""
Sep 10 15:59:03.385223 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.385195    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"]
Sep 10 15:59:03.409566 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.409524    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-podinfo\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.409731 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.409579    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istiod-ca-cert\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.409731 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.409627    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57kfs\" (UniqueName: \"kubernetes.io/projected/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-kube-api-access-57kfs\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.409731 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.409672    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-token\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.409731 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.409702    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-workload-socket\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.409731 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.409728    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-envoy\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.409976 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.409787    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-workload-certs\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.409976 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.409826    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-data\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.409976 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.409872    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-credential-socket\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.510495 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.510459    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-token\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.510495 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.510504    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-workload-socket\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.510707 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.510528    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-envoy\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.510707 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.510571    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-workload-certs\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.510707 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.510595    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-data\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.510707 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.510644    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-credential-socket\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.510707 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.510688    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-podinfo\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.510958 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.510725    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istiod-ca-cert\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.510958 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.510837    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57kfs\" (UniqueName: \"kubernetes.io/projected/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-kube-api-access-57kfs\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.511144 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.510980    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-workload-socket\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.511144 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.511031    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-workload-certs\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.511296 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.511194    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-data\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.511296 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.511214    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-credential-socket\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.511545 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.511513    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istiod-ca-cert\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.513048 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.513025    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-envoy\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.513348 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.513327    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-podinfo\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.519612 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.519591    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-token\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.519785 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.519768    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57kfs\" (UniqueName: \"kubernetes.io/projected/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-kube-api-access-57kfs\") pod \"e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") " pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.675018 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.674990    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:03.958025 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.957993    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"]
Sep 10 15:59:03.959863 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:59:03.959821    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb61d2eb_8d3f_49e9_8d12_8a34abbbb93b.slice/crio-cd6eda0f5a3e82c1dfa543b79ebeea206d9167efe6c6d69b802f96103ed0a7c9 WatchSource:0}: Error finding container cd6eda0f5a3e82c1dfa543b79ebeea206d9167efe6c6d69b802f96103ed0a7c9: Status 404 returned error can't find the container with id cd6eda0f5a3e82c1dfa543b79ebeea206d9167efe6c6d69b802f96103ed0a7c9
Sep 10 15:59:03.962242 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.962207    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:59:03.962373 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.962284    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:59:03.962373 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:03.962312    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:59:04.207883 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:04.207786    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7" event={"ID":"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b","Type":"ContainerStarted","Data":"807a45ea25c10ea8a6e8786fc9db5025bd41696d75a596efd996b1e29faaf4c5"}
Sep 10 15:59:04.207883 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:04.207842    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7" event={"ID":"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b","Type":"ContainerStarted","Data":"cd6eda0f5a3e82c1dfa543b79ebeea206d9167efe6c6d69b802f96103ed0a7c9"}
Sep 10 15:59:04.235130 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:04.235065    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7" podStartSLOduration=1.235042483 podStartE2EDuration="1.235042483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:59:04.231830031 +0000 UTC m=+1023.036149292" watchObservedRunningTime="2026-09-10 15:59:04.235042483 +0000 UTC m=+1023.039361744"
Sep 10 15:59:04.675255 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:04.675220    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:05.680418 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:05.680385    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:06.219976 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:06.219939    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:06.221111 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:06.221091    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:26.354611 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:26.354579    2564 generic.go:356] "Generic (PLEG): container finished" podID="fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" containerID="807a45ea25c10ea8a6e8786fc9db5025bd41696d75a596efd996b1e29faaf4c5" exitCode=0
Sep 10 15:59:26.355056 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:26.354654    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7" event={"ID":"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b","Type":"ContainerDied","Data":"807a45ea25c10ea8a6e8786fc9db5025bd41696d75a596efd996b1e29faaf4c5"}
Sep 10 15:59:26.355135 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:26.355103    2564 scope.go:117] "RemoveContainer" containerID="807a45ea25c10ea8a6e8786fc9db5025bd41696d75a596efd996b1e29faaf4c5"
Sep 10 15:59:26.356402 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:26.356372    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:59:26.356481 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:26.356446    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:59:26.356521 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:26.356478    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:59:27.363939 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:27.363905    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7" event={"ID":"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b","Type":"ContainerStarted","Data":"8ec3def874b5ee908b09bfd5548756475ca117b5fd003055a4f0c8c4bcd55f97"}
Sep 10 15:59:27.675522 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:27.675492    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:27.675755 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:27.675732    2564 patch_prober.go:28] interesting pod/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 10 15:59:27.675852 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:27.675797    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7" podUID="fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 10 15:59:28.675586 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:28.675544    2564 patch_prober.go:28] interesting pod/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 10 15:59:28.676071 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:28.675612    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7" podUID="fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 10 15:59:29.113032 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:29.112946    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"]
Sep 10 15:59:29.675977 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:29.675924    2564 patch_prober.go:28] interesting pod/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 10 15:59:29.676406 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:29.675992    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7" podUID="fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 10 15:59:30.675909 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:30.675835    2564 patch_prober.go:28] interesting pod/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 10 15:59:30.676110 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:30.675965    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7" podUID="fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 10 15:59:31.108302 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.108199    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bbc679795-dzhmg"]
Sep 10 15:59:31.108945 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.108919    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ed3c88-bfeb-4286-a429-e0d406cf6931" containerName="istio-proxy"
Sep 10 15:59:31.108945 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.108945    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ed3c88-bfeb-4286-a429-e0d406cf6931" containerName="istio-proxy"
Sep 10 15:59:31.109147 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.109070    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="97ed3c88-bfeb-4286-a429-e0d406cf6931" containerName="istio-proxy"
Sep 10 15:59:31.114138 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.114110    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.132623 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.132587    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bbc679795-dzhmg"]
Sep 10 15:59:31.215185 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.214842    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3989901e-a627-4be6-bbde-dfa31a067064-console-oauth-config\") pod \"console-6bbc679795-dzhmg\" (UID: \"3989901e-a627-4be6-bbde-dfa31a067064\") " pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.215185 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.214951    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3989901e-a627-4be6-bbde-dfa31a067064-trusted-ca-bundle\") pod \"console-6bbc679795-dzhmg\" (UID: \"3989901e-a627-4be6-bbde-dfa31a067064\") " pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.215185 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.215020    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtkgz\" (UniqueName: \"kubernetes.io/projected/3989901e-a627-4be6-bbde-dfa31a067064-kube-api-access-dtkgz\") pod \"console-6bbc679795-dzhmg\" (UID: \"3989901e-a627-4be6-bbde-dfa31a067064\") " pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.215185 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.215092    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3989901e-a627-4be6-bbde-dfa31a067064-console-serving-cert\") pod \"console-6bbc679795-dzhmg\" (UID: \"3989901e-a627-4be6-bbde-dfa31a067064\") " pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.215185 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.215195    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3989901e-a627-4be6-bbde-dfa31a067064-service-ca\") pod \"console-6bbc679795-dzhmg\" (UID: \"3989901e-a627-4be6-bbde-dfa31a067064\") " pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.215647 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.215284    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3989901e-a627-4be6-bbde-dfa31a067064-console-config\") pod \"console-6bbc679795-dzhmg\" (UID: \"3989901e-a627-4be6-bbde-dfa31a067064\") " pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.215647 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.215331    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3989901e-a627-4be6-bbde-dfa31a067064-oauth-serving-cert\") pod \"console-6bbc679795-dzhmg\" (UID: \"3989901e-a627-4be6-bbde-dfa31a067064\") " pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.316181 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.316122    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3989901e-a627-4be6-bbde-dfa31a067064-console-serving-cert\") pod \"console-6bbc679795-dzhmg\" (UID: \"3989901e-a627-4be6-bbde-dfa31a067064\") " pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.316370 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.316215    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3989901e-a627-4be6-bbde-dfa31a067064-service-ca\") pod \"console-6bbc679795-dzhmg\" (UID: \"3989901e-a627-4be6-bbde-dfa31a067064\") " pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.316370 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.316272    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3989901e-a627-4be6-bbde-dfa31a067064-console-config\") pod \"console-6bbc679795-dzhmg\" (UID: \"3989901e-a627-4be6-bbde-dfa31a067064\") " pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.316370 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.316299    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3989901e-a627-4be6-bbde-dfa31a067064-oauth-serving-cert\") pod \"console-6bbc679795-dzhmg\" (UID: \"3989901e-a627-4be6-bbde-dfa31a067064\") " pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.316572 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.316424    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3989901e-a627-4be6-bbde-dfa31a067064-console-oauth-config\") pod \"console-6bbc679795-dzhmg\" (UID: \"3989901e-a627-4be6-bbde-dfa31a067064\") " pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.316572 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.316456    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3989901e-a627-4be6-bbde-dfa31a067064-trusted-ca-bundle\") pod \"console-6bbc679795-dzhmg\" (UID: \"3989901e-a627-4be6-bbde-dfa31a067064\") " pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.316572 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.316484    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtkgz\" (UniqueName: \"kubernetes.io/projected/3989901e-a627-4be6-bbde-dfa31a067064-kube-api-access-dtkgz\") pod \"console-6bbc679795-dzhmg\" (UID: \"3989901e-a627-4be6-bbde-dfa31a067064\") " pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.318035 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.318007    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3989901e-a627-4be6-bbde-dfa31a067064-service-ca\") pod \"console-6bbc679795-dzhmg\" (UID: \"3989901e-a627-4be6-bbde-dfa31a067064\") " pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.318216 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.318023    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3989901e-a627-4be6-bbde-dfa31a067064-oauth-serving-cert\") pod \"console-6bbc679795-dzhmg\" (UID: \"3989901e-a627-4be6-bbde-dfa31a067064\") " pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.318916 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.318895    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3989901e-a627-4be6-bbde-dfa31a067064-console-config\") pod \"console-6bbc679795-dzhmg\" (UID: \"3989901e-a627-4be6-bbde-dfa31a067064\") " pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.319186 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.319117    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3989901e-a627-4be6-bbde-dfa31a067064-trusted-ca-bundle\") pod \"console-6bbc679795-dzhmg\" (UID: \"3989901e-a627-4be6-bbde-dfa31a067064\") " pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.323665 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.323620    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3989901e-a627-4be6-bbde-dfa31a067064-console-oauth-config\") pod \"console-6bbc679795-dzhmg\" (UID: \"3989901e-a627-4be6-bbde-dfa31a067064\") " pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.324281 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.324245    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3989901e-a627-4be6-bbde-dfa31a067064-console-serving-cert\") pod \"console-6bbc679795-dzhmg\" (UID: \"3989901e-a627-4be6-bbde-dfa31a067064\") " pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.327534 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.327515    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtkgz\" (UniqueName: \"kubernetes.io/projected/3989901e-a627-4be6-bbde-dfa31a067064-kube-api-access-dtkgz\") pod \"console-6bbc679795-dzhmg\" (UID: \"3989901e-a627-4be6-bbde-dfa31a067064\") " pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.432390 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.432350    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:31.676131 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.676097    2564 patch_prober.go:28] interesting pod/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 10 15:59:31.676619 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.676146    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7" podUID="fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 10 15:59:31.734406 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:31.734337    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bbc679795-dzhmg"]
Sep 10 15:59:31.736835 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:59:31.736805    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3989901e_a627_4be6_bbde_dfa31a067064.slice/crio-31468f4c25c48916785e84b42eb50702bf5d495711c5198013cdf59ee9b2a700 WatchSource:0}: Error finding container 31468f4c25c48916785e84b42eb50702bf5d495711c5198013cdf59ee9b2a700: Status 404 returned error can't find the container with id 31468f4c25c48916785e84b42eb50702bf5d495711c5198013cdf59ee9b2a700
Sep 10 15:59:32.400364 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:32.400313    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bbc679795-dzhmg" event={"ID":"3989901e-a627-4be6-bbde-dfa31a067064","Type":"ContainerStarted","Data":"0be1dfd6a7850a72acce481947d1b2f564e1e1fca8a3d0f362dee7f8016c49e6"}
Sep 10 15:59:32.400553 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:32.400392    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bbc679795-dzhmg" event={"ID":"3989901e-a627-4be6-bbde-dfa31a067064","Type":"ContainerStarted","Data":"31468f4c25c48916785e84b42eb50702bf5d495711c5198013cdf59ee9b2a700"}
Sep 10 15:59:32.424222 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:32.424148    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bbc679795-dzhmg" podStartSLOduration=1.424131587 podStartE2EDuration="1.424131587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:59:32.42061704 +0000 UTC m=+1051.224936302" watchObservedRunningTime="2026-09-10 15:59:32.424131587 +0000 UTC m=+1051.228450885"
Sep 10 15:59:32.729756 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:32.729725    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:32.730211 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:32.730053    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:32.730918 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:32.730896    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 15:59:39.154057 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:39.154023    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a70cb1a-dxz5m"]
Sep 10 15:59:39.164846 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:39.164815    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a70cb1a-dxz5m"
Sep 10 15:59:39.168311 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:39.168048    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2wghg\""
Sep 10 15:59:39.169642 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:39.169614    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a70cb1a-dxz5m"]
Sep 10 15:59:39.203459 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:39.203420    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47f1386e-4577-43a5-92c7-c08e9259fa9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9a70cb1a-dxz5m\" (UID: \"47f1386e-4577-43a5-92c7-c08e9259fa9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a70cb1a-dxz5m"
Sep 10 15:59:39.304338 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:39.304303    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47f1386e-4577-43a5-92c7-c08e9259fa9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9a70cb1a-dxz5m\" (UID: \"47f1386e-4577-43a5-92c7-c08e9259fa9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a70cb1a-dxz5m"
Sep 10 15:59:39.304975 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:39.304949    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47f1386e-4577-43a5-92c7-c08e9259fa9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9a70cb1a-dxz5m\" (UID: \"47f1386e-4577-43a5-92c7-c08e9259fa9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a70cb1a-dxz5m"
Sep 10 15:59:39.479640 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:39.479609    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a70cb1a-dxz5m"
Sep 10 15:59:39.759712 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:39.759625    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a70cb1a-dxz5m"]
Sep 10 15:59:39.763279 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:59:39.763242    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f1386e_4577_43a5_92c7_c08e9259fa9a.slice/crio-5cb284a1b423c457a513b0e0a6bd3d352a08f88856d8101a57952994c089a784 WatchSource:0}: Error finding container 5cb284a1b423c457a513b0e0a6bd3d352a08f88856d8101a57952994c089a784: Status 404 returned error can't find the container with id 5cb284a1b423c457a513b0e0a6bd3d352a08f88856d8101a57952994c089a784
Sep 10 15:59:40.450595 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:40.450560    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a70cb1a-dxz5m" event={"ID":"47f1386e-4577-43a5-92c7-c08e9259fa9a","Type":"ContainerStarted","Data":"0ab9d40477235c7aef6f3bb0bf76dc58df35ab85548ccf7c0a62dd3690e0c827"}
Sep 10 15:59:40.451000 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:40.450617    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a70cb1a-dxz5m" event={"ID":"47f1386e-4577-43a5-92c7-c08e9259fa9a","Type":"ContainerStarted","Data":"5cb284a1b423c457a513b0e0a6bd3d352a08f88856d8101a57952994c089a784"}
Sep 10 15:59:40.470467 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:40.470416    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a70cb1a-dxz5m" podStartSLOduration=1.4704013470000001 podStartE2EDuration="1.470401347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:59:40.468363894 +0000 UTC m=+1059.272683161" watchObservedRunningTime="2026-09-10 15:59:40.470401347 +0000 UTC m=+1059.274720607"
Sep 10 15:59:41.433229 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:41.433187    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:41.433416 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:41.433295    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:41.438151 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:41.438127    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:41.457649 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:41.457622    2564 generic.go:356] "Generic (PLEG): container finished" podID="47f1386e-4577-43a5-92c7-c08e9259fa9a" containerID="0ab9d40477235c7aef6f3bb0bf76dc58df35ab85548ccf7c0a62dd3690e0c827" exitCode=0
Sep 10 15:59:41.458179 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:41.457714    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a70cb1a-dxz5m" event={"ID":"47f1386e-4577-43a5-92c7-c08e9259fa9a","Type":"ContainerDied","Data":"0ab9d40477235c7aef6f3bb0bf76dc58df35ab85548ccf7c0a62dd3690e0c827"}
Sep 10 15:59:41.462002 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:41.461978    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bbc679795-dzhmg"
Sep 10 15:59:41.531386 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:41.531357    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-678d855bc7-cgqgg"]
Sep 10 15:59:42.608623 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:42.608593    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a70cb1a-dxz5m"
Sep 10 15:59:42.640863 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:42.640828    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47f1386e-4577-43a5-92c7-c08e9259fa9a-maas-api-service-ca\") pod \"47f1386e-4577-43a5-92c7-c08e9259fa9a\" (UID: \"47f1386e-4577-43a5-92c7-c08e9259fa9a\") "
Sep 10 15:59:42.641495 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:42.641460    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f1386e-4577-43a5-92c7-c08e9259fa9a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "47f1386e-4577-43a5-92c7-c08e9259fa9a" (UID: "47f1386e-4577-43a5-92c7-c08e9259fa9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:59:42.641601 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:42.641586    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47f1386e-4577-43a5-92c7-c08e9259fa9a-maas-api-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:59:42.680206 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:42.680174    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-83dcaf-pkcrr"]
Sep 10 15:59:42.680623 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:42.680611    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47f1386e-4577-43a5-92c7-c08e9259fa9a" containerName="revoke-keys"
Sep 10 15:59:42.680668 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:42.680624    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="47f1386e-4577-43a5-92c7-c08e9259fa9a" containerName="revoke-keys"
Sep 10 15:59:42.680719 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:42.680709    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="47f1386e-4577-43a5-92c7-c08e9259fa9a" containerName="revoke-keys"
Sep 10 15:59:42.683854 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:42.683835    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-83dcaf-pkcrr"
Sep 10 15:59:42.689798 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:42.689773    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-83dcaf-pkcrr"]
Sep 10 15:59:42.742333 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:42.742231    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b86509f-3dd3-4b0b-aaff-4b82565113f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-83dcaf-pkcrr\" (UID: \"7b86509f-3dd3-4b0b-aaff-4b82565113f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-83dcaf-pkcrr"
Sep 10 15:59:42.842947 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:42.842914    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b86509f-3dd3-4b0b-aaff-4b82565113f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-83dcaf-pkcrr\" (UID: \"7b86509f-3dd3-4b0b-aaff-4b82565113f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-83dcaf-pkcrr"
Sep 10 15:59:42.843599 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:42.843572    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b86509f-3dd3-4b0b-aaff-4b82565113f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-83dcaf-pkcrr\" (UID: \"7b86509f-3dd3-4b0b-aaff-4b82565113f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-83dcaf-pkcrr"
Sep 10 15:59:42.996350 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:42.996257    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-83dcaf-pkcrr"
Sep 10 15:59:43.281814 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:43.281787    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-83dcaf-pkcrr"]
Sep 10 15:59:43.283848 ip-10-0-128-145 kubenswrapper[2564]: W0910 15:59:43.283816    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b86509f_3dd3_4b0b_aaff_4b82565113f3.slice/crio-a917f374b2f1fa9690a12029149963884209716e0b40842eb75642d14f153c6f WatchSource:0}: Error finding container a917f374b2f1fa9690a12029149963884209716e0b40842eb75642d14f153c6f: Status 404 returned error can't find the container with id a917f374b2f1fa9690a12029149963884209716e0b40842eb75642d14f153c6f
Sep 10 15:59:43.470434 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:43.470408    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a70cb1a-dxz5m"
Sep 10 15:59:43.470621 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:43.470408    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a70cb1a-dxz5m" event={"ID":"47f1386e-4577-43a5-92c7-c08e9259fa9a","Type":"ContainerDied","Data":"5cb284a1b423c457a513b0e0a6bd3d352a08f88856d8101a57952994c089a784"}
Sep 10 15:59:43.470621 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:43.470534    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cb284a1b423c457a513b0e0a6bd3d352a08f88856d8101a57952994c089a784"
Sep 10 15:59:43.471766 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:43.471740    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-83dcaf-pkcrr" event={"ID":"7b86509f-3dd3-4b0b-aaff-4b82565113f3","Type":"ContainerStarted","Data":"a917f374b2f1fa9690a12029149963884209716e0b40842eb75642d14f153c6f"}
Sep 10 15:59:44.159640 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.159600    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b" podUID="de68757c-acc3-4495-b69b-4e05cff36df9" containerName="main" containerID="cri-o://52dae10ccfd7307feadac9410f176f469d8e471dc6bc8a4c12ba9e22ab4877f9" gracePeriod=45
Sep 10 15:59:44.417374 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.417347    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:59:44.460549 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.460514    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-model-cache\") pod \"de68757c-acc3-4495-b69b-4e05cff36df9\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") "
Sep 10 15:59:44.460733 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.460558    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94qtx\" (UniqueName: \"kubernetes.io/projected/de68757c-acc3-4495-b69b-4e05cff36df9-kube-api-access-94qtx\") pod \"de68757c-acc3-4495-b69b-4e05cff36df9\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") "
Sep 10 15:59:44.460733 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.460636    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-home\") pod \"de68757c-acc3-4495-b69b-4e05cff36df9\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") "
Sep 10 15:59:44.460733 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.460676    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-dshm\") pod \"de68757c-acc3-4495-b69b-4e05cff36df9\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") "
Sep 10 15:59:44.460733 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.460721    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-tmp-dir\") pod \"de68757c-acc3-4495-b69b-4e05cff36df9\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") "
Sep 10 15:59:44.460971 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.460772    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de68757c-acc3-4495-b69b-4e05cff36df9-tls-certs\") pod \"de68757c-acc3-4495-b69b-4e05cff36df9\" (UID: \"de68757c-acc3-4495-b69b-4e05cff36df9\") "
Sep 10 15:59:44.460971 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.460789    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-model-cache" (OuterVolumeSpecName: "model-cache") pod "de68757c-acc3-4495-b69b-4e05cff36df9" (UID: "de68757c-acc3-4495-b69b-4e05cff36df9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:44.461086 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.460976    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de68757c-acc3-4495-b69b-4e05cff36df9" (UID: "de68757c-acc3-4495-b69b-4e05cff36df9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:44.461189 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.461148    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-model-cache\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:59:44.461189 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.461181    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-tmp-dir\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:59:44.461554 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.461529    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-home" (OuterVolumeSpecName: "home") pod "de68757c-acc3-4495-b69b-4e05cff36df9" (UID: "de68757c-acc3-4495-b69b-4e05cff36df9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:44.463112 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.463083    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de68757c-acc3-4495-b69b-4e05cff36df9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de68757c-acc3-4495-b69b-4e05cff36df9" (UID: "de68757c-acc3-4495-b69b-4e05cff36df9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:59:44.463275 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.463134    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-dshm" (OuterVolumeSpecName: "dshm") pod "de68757c-acc3-4495-b69b-4e05cff36df9" (UID: "de68757c-acc3-4495-b69b-4e05cff36df9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:44.463348 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.463332    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de68757c-acc3-4495-b69b-4e05cff36df9-kube-api-access-94qtx" (OuterVolumeSpecName: "kube-api-access-94qtx") pod "de68757c-acc3-4495-b69b-4e05cff36df9" (UID: "de68757c-acc3-4495-b69b-4e05cff36df9"). InnerVolumeSpecName "kube-api-access-94qtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:59:44.479959 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.479934    2564 generic.go:356] "Generic (PLEG): container finished" podID="7b86509f-3dd3-4b0b-aaff-4b82565113f3" containerID="9ffb4fb4d90b9cb8fe799eca00aba7502142e04794cf811296a23f64ebce8b3e" exitCode=0
Sep 10 15:59:44.480077 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.480021    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-83dcaf-pkcrr" event={"ID":"7b86509f-3dd3-4b0b-aaff-4b82565113f3","Type":"ContainerDied","Data":"9ffb4fb4d90b9cb8fe799eca00aba7502142e04794cf811296a23f64ebce8b3e"}
Sep 10 15:59:44.481315 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.481296    2564 generic.go:356] "Generic (PLEG): container finished" podID="de68757c-acc3-4495-b69b-4e05cff36df9" containerID="52dae10ccfd7307feadac9410f176f469d8e471dc6bc8a4c12ba9e22ab4877f9" exitCode=0
Sep 10 15:59:44.481415 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.481333    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"
Sep 10 15:59:44.481415 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.481381    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b" event={"ID":"de68757c-acc3-4495-b69b-4e05cff36df9","Type":"ContainerDied","Data":"52dae10ccfd7307feadac9410f176f469d8e471dc6bc8a4c12ba9e22ab4877f9"}
Sep 10 15:59:44.481519 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.481421    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b" event={"ID":"de68757c-acc3-4495-b69b-4e05cff36df9","Type":"ContainerDied","Data":"7194839c2082727d4c16977c00bba7775a9ae7d7743d9b8bcebbd51df2b69a1c"}
Sep 10 15:59:44.481519 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.481445    2564 scope.go:117] "RemoveContainer" containerID="52dae10ccfd7307feadac9410f176f469d8e471dc6bc8a4c12ba9e22ab4877f9"
Sep 10 15:59:44.491014 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.490994    2564 scope.go:117] "RemoveContainer" containerID="52dae10ccfd7307feadac9410f176f469d8e471dc6bc8a4c12ba9e22ab4877f9"
Sep 10 15:59:44.491382 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:59:44.491363    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52dae10ccfd7307feadac9410f176f469d8e471dc6bc8a4c12ba9e22ab4877f9\": container with ID starting with 52dae10ccfd7307feadac9410f176f469d8e471dc6bc8a4c12ba9e22ab4877f9 not found: ID does not exist" containerID="52dae10ccfd7307feadac9410f176f469d8e471dc6bc8a4c12ba9e22ab4877f9"
Sep 10 15:59:44.491442 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.491391    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52dae10ccfd7307feadac9410f176f469d8e471dc6bc8a4c12ba9e22ab4877f9"} err="failed to get container status \"52dae10ccfd7307feadac9410f176f469d8e471dc6bc8a4c12ba9e22ab4877f9\": rpc error: code = NotFound desc = could not find container \"52dae10ccfd7307feadac9410f176f469d8e471dc6bc8a4c12ba9e22ab4877f9\": container with ID starting with 52dae10ccfd7307feadac9410f176f469d8e471dc6bc8a4c12ba9e22ab4877f9 not found: ID does not exist"
Sep 10 15:59:44.513494 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.513466    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"]
Sep 10 15:59:44.517949 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.517926    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-83dcaf/test-model-83dcaf-kserve-788df77fff-7sc5b"]
Sep 10 15:59:44.562396 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.562368    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de68757c-acc3-4495-b69b-4e05cff36df9-tls-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:59:44.562396 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.562393    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94qtx\" (UniqueName: \"kubernetes.io/projected/de68757c-acc3-4495-b69b-4e05cff36df9-kube-api-access-94qtx\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:59:44.562396 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.562402    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-home\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:59:44.562610 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:44.562413    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de68757c-acc3-4495-b69b-4e05cff36df9-dshm\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:59:45.647539 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:45.647510    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-83dcaf-pkcrr"
Sep 10 15:59:45.677219 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:45.677185    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de68757c-acc3-4495-b69b-4e05cff36df9" path="/var/lib/kubelet/pods/de68757c-acc3-4495-b69b-4e05cff36df9/volumes"
Sep 10 15:59:45.774788 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:45.774699    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b86509f-3dd3-4b0b-aaff-4b82565113f3-maas-api-service-ca\") pod \"7b86509f-3dd3-4b0b-aaff-4b82565113f3\" (UID: \"7b86509f-3dd3-4b0b-aaff-4b82565113f3\") "
Sep 10 15:59:45.775049 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:45.775028    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b86509f-3dd3-4b0b-aaff-4b82565113f3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b86509f-3dd3-4b0b-aaff-4b82565113f3" (UID: "7b86509f-3dd3-4b0b-aaff-4b82565113f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:59:45.876058 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:45.876019    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b86509f-3dd3-4b0b-aaff-4b82565113f3-maas-api-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:59:46.500522 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:46.500494    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-83dcaf-pkcrr"
Sep 10 15:59:46.500719 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:46.500496    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-83dcaf-pkcrr" event={"ID":"7b86509f-3dd3-4b0b-aaff-4b82565113f3","Type":"ContainerDied","Data":"a917f374b2f1fa9690a12029149963884209716e0b40842eb75642d14f153c6f"}
Sep 10 15:59:46.500719 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:46.500661    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a917f374b2f1fa9690a12029149963884209716e0b40842eb75642d14f153c6f"
Sep 10 15:59:50.456737 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:50.456698    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a70cb1a-dxz5m"]
Sep 10 15:59:50.463620 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:50.463593    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9a70cb1a-dxz5m"]
Sep 10 15:59:51.673290 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:51.673258    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47f1386e-4577-43a5-92c7-c08e9259fa9a" path="/var/lib/kubelet/pods/47f1386e-4577-43a5-92c7-c08e9259fa9a/volumes"
Sep 10 15:59:53.626562 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:53.626480    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v"]
Sep 10 15:59:53.626964 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:53.626757    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v" podUID="da45bba5-1d5b-4a1c-8ff2-61bead26ab19" containerName="payload-processing" containerID="cri-o://0609285d9012e47d49445f76f6e3ffa5444f9fd3cbe7fceb1493e8aa7ee3b877" gracePeriod=30
Sep 10 15:59:53.805249 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:53.805221    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-83dcaf-pkcrr"]
Sep 10 15:59:53.807683 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:53.807659    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-83dcaf-pkcrr"]
Sep 10 15:59:53.894900 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:53.894831    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v"
Sep 10 15:59:53.961050 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:53.961025    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25llz\" (UniqueName: \"kubernetes.io/projected/da45bba5-1d5b-4a1c-8ff2-61bead26ab19-kube-api-access-25llz\") pod \"da45bba5-1d5b-4a1c-8ff2-61bead26ab19\" (UID: \"da45bba5-1d5b-4a1c-8ff2-61bead26ab19\") "
Sep 10 15:59:53.961274 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:53.961144    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da45bba5-1d5b-4a1c-8ff2-61bead26ab19-plugins-config-volume\") pod \"da45bba5-1d5b-4a1c-8ff2-61bead26ab19\" (UID: \"da45bba5-1d5b-4a1c-8ff2-61bead26ab19\") "
Sep 10 15:59:53.961521 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:53.961493    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da45bba5-1d5b-4a1c-8ff2-61bead26ab19-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "da45bba5-1d5b-4a1c-8ff2-61bead26ab19" (UID: "da45bba5-1d5b-4a1c-8ff2-61bead26ab19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:59:53.963221 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:53.963200    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da45bba5-1d5b-4a1c-8ff2-61bead26ab19-kube-api-access-25llz" (OuterVolumeSpecName: "kube-api-access-25llz") pod "da45bba5-1d5b-4a1c-8ff2-61bead26ab19" (UID: "da45bba5-1d5b-4a1c-8ff2-61bead26ab19"). InnerVolumeSpecName "kube-api-access-25llz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:59:54.062628 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:54.062590    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25llz\" (UniqueName: \"kubernetes.io/projected/da45bba5-1d5b-4a1c-8ff2-61bead26ab19-kube-api-access-25llz\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:59:54.062628 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:54.062622    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da45bba5-1d5b-4a1c-8ff2-61bead26ab19-plugins-config-volume\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 15:59:54.555150 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:54.555116    2564 generic.go:356] "Generic (PLEG): container finished" podID="da45bba5-1d5b-4a1c-8ff2-61bead26ab19" containerID="0609285d9012e47d49445f76f6e3ffa5444f9fd3cbe7fceb1493e8aa7ee3b877" exitCode=0
Sep 10 15:59:54.555351 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:54.555186    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v"
Sep 10 15:59:54.555351 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:54.555203    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v" event={"ID":"da45bba5-1d5b-4a1c-8ff2-61bead26ab19","Type":"ContainerDied","Data":"0609285d9012e47d49445f76f6e3ffa5444f9fd3cbe7fceb1493e8aa7ee3b877"}
Sep 10 15:59:54.555351 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:54.555258    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v" event={"ID":"da45bba5-1d5b-4a1c-8ff2-61bead26ab19","Type":"ContainerDied","Data":"13a78dfe1c1bc11da131a4d9ade124a71dc6e268049539a47837e51c3b92b91e"}
Sep 10 15:59:54.555351 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:54.555280    2564 scope.go:117] "RemoveContainer" containerID="0609285d9012e47d49445f76f6e3ffa5444f9fd3cbe7fceb1493e8aa7ee3b877"
Sep 10 15:59:54.564951 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:54.564933    2564 scope.go:117] "RemoveContainer" containerID="0609285d9012e47d49445f76f6e3ffa5444f9fd3cbe7fceb1493e8aa7ee3b877"
Sep 10 15:59:54.565252 ip-10-0-128-145 kubenswrapper[2564]: E0910 15:59:54.565236    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0609285d9012e47d49445f76f6e3ffa5444f9fd3cbe7fceb1493e8aa7ee3b877\": container with ID starting with 0609285d9012e47d49445f76f6e3ffa5444f9fd3cbe7fceb1493e8aa7ee3b877 not found: ID does not exist" containerID="0609285d9012e47d49445f76f6e3ffa5444f9fd3cbe7fceb1493e8aa7ee3b877"
Sep 10 15:59:54.565306 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:54.565260    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0609285d9012e47d49445f76f6e3ffa5444f9fd3cbe7fceb1493e8aa7ee3b877"} err="failed to get container status \"0609285d9012e47d49445f76f6e3ffa5444f9fd3cbe7fceb1493e8aa7ee3b877\": rpc error: code = NotFound desc = could not find container \"0609285d9012e47d49445f76f6e3ffa5444f9fd3cbe7fceb1493e8aa7ee3b877\": container with ID starting with 0609285d9012e47d49445f76f6e3ffa5444f9fd3cbe7fceb1493e8aa7ee3b877 not found: ID does not exist"
Sep 10 15:59:54.580305 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:54.580269    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v"]
Sep 10 15:59:54.582349 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:54.582330    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-83dcaf-8c9c5ddc9-m5b4v"]
Sep 10 15:59:55.673352 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:55.673322    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b86509f-3dd3-4b0b-aaff-4b82565113f3" path="/var/lib/kubelet/pods/7b86509f-3dd3-4b0b-aaff-4b82565113f3/volumes"
Sep 10 15:59:55.673717 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:55.673686    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da45bba5-1d5b-4a1c-8ff2-61bead26ab19" path="/var/lib/kubelet/pods/da45bba5-1d5b-4a1c-8ff2-61bead26ab19/volumes"
Sep 10 15:59:57.655969 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:57.655932    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"]
Sep 10 15:59:57.656368 ip-10-0-128-145 kubenswrapper[2564]: I0910 15:59:57.656257    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7" podUID="fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" containerName="istio-proxy" containerID="cri-o://8ec3def874b5ee908b09bfd5548756475ca117b5fd003055a4f0c8c4bcd55f97" gracePeriod=30
Sep 10 16:00:00.164376 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:00.164342    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817600-6bb9m"]
Sep 10 16:00:00.164962 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:00.164941    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b86509f-3dd3-4b0b-aaff-4b82565113f3" containerName="revoke-keys"
Sep 10 16:00:00.165063 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:00.164964    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b86509f-3dd3-4b0b-aaff-4b82565113f3" containerName="revoke-keys"
Sep 10 16:00:00.165063 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:00.164982    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da45bba5-1d5b-4a1c-8ff2-61bead26ab19" containerName="payload-processing"
Sep 10 16:00:00.165063 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:00.164989    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="da45bba5-1d5b-4a1c-8ff2-61bead26ab19" containerName="payload-processing"
Sep 10 16:00:00.165063 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:00.165022    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de68757c-acc3-4495-b69b-4e05cff36df9" containerName="main"
Sep 10 16:00:00.165063 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:00.165031    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="de68757c-acc3-4495-b69b-4e05cff36df9" containerName="main"
Sep 10 16:00:00.165326 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:00.165139    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b86509f-3dd3-4b0b-aaff-4b82565113f3" containerName="revoke-keys"
Sep 10 16:00:00.165326 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:00.165177    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="da45bba5-1d5b-4a1c-8ff2-61bead26ab19" containerName="payload-processing"
Sep 10 16:00:00.165326 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:00.165193    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="de68757c-acc3-4495-b69b-4e05cff36df9" containerName="main"
Sep 10 16:00:00.168298 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:00.168279    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-6bb9m"
Sep 10 16:00:00.171333 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:00.171309    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2wghg\""
Sep 10 16:00:00.179511 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:00.179486    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-6bb9m"
Sep 10 16:00:00.179756 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:00.179734    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817600-6bb9m"]
Sep 10 16:00:00.465481 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:00.465423    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817600-6bb9m"]
Sep 10 16:00:00.468031 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:00:00.468005    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34637ef5_80be_4d7b_ad69_cfeacc663e43.slice/crio-26ec1f03ab49275e3dc5102a7e0269fb30a3d5f0bd02d27f619f42c6ec5f854b WatchSource:0}: Error finding container 26ec1f03ab49275e3dc5102a7e0269fb30a3d5f0bd02d27f619f42c6ec5f854b: Status 404 returned error can't find the container with id 26ec1f03ab49275e3dc5102a7e0269fb30a3d5f0bd02d27f619f42c6ec5f854b
Sep 10 16:00:00.599447 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:00.599402    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-6bb9m" event={"ID":"34637ef5-80be-4d7b-ad69-cfeacc663e43","Type":"ContainerStarted","Data":"26ec1f03ab49275e3dc5102a7e0269fb30a3d5f0bd02d27f619f42c6ec5f854b"}
Sep 10 16:00:01.525787 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:01.525754    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"]
Sep 10 16:00:01.526226 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:01.526012    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg" podUID="3765fb24-410e-4107-856c-af41b073836b" containerName="istio-proxy" containerID="cri-o://2318776fc92194de041ed8b497306e3161c6da995615200c94fce80d7b08d5d0" gracePeriod=30
Sep 10 16:00:01.607430 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:01.607400    2564 generic.go:356] "Generic (PLEG): container finished" podID="34637ef5-80be-4d7b-ad69-cfeacc663e43" containerID="5183af843022f5fb308575df7f24e657b4565db14738067504fbea8d96ce0bd5" exitCode=0
Sep 10 16:00:01.607602 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:01.607473    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-6bb9m" event={"ID":"34637ef5-80be-4d7b-ad69-cfeacc663e43","Type":"ContainerDied","Data":"5183af843022f5fb308575df7f24e657b4565db14738067504fbea8d96ce0bd5"}
Sep 10 16:00:01.923737 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:01.923701    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-607b9a-6c7x8"]
Sep 10 16:00:01.928202 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:01.928180    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-607b9a-6c7x8"
Sep 10 16:00:01.939458 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:01.939427    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-607b9a-6c7x8"]
Sep 10 16:00:02.047125 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:02.047085    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d749003-c1d1-4b55-9e6f-2930a387b05f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-607b9a-6c7x8\" (UID: \"6d749003-c1d1-4b55-9e6f-2930a387b05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-607b9a-6c7x8"
Sep 10 16:00:02.148437 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:02.148395    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d749003-c1d1-4b55-9e6f-2930a387b05f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-607b9a-6c7x8\" (UID: \"6d749003-c1d1-4b55-9e6f-2930a387b05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-607b9a-6c7x8"
Sep 10 16:00:02.149015 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:02.148995    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d749003-c1d1-4b55-9e6f-2930a387b05f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-607b9a-6c7x8\" (UID: \"6d749003-c1d1-4b55-9e6f-2930a387b05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-607b9a-6c7x8"
Sep 10 16:00:02.239901 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:02.239809    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-607b9a-6c7x8"
Sep 10 16:00:02.517178 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:02.517135    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-607b9a-6c7x8"]
Sep 10 16:00:02.518851 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:00:02.518828    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d749003_c1d1_4b55_9e6f_2930a387b05f.slice/crio-1d34d22d6ebae088d70f97db00d19c736cb78bac2aa56e828979f4433194968b WatchSource:0}: Error finding container 1d34d22d6ebae088d70f97db00d19c736cb78bac2aa56e828979f4433194968b: Status 404 returned error can't find the container with id 1d34d22d6ebae088d70f97db00d19c736cb78bac2aa56e828979f4433194968b
Sep 10 16:00:02.615247 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:02.615214    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-607b9a-6c7x8" event={"ID":"6d749003-c1d1-4b55-9e6f-2930a387b05f","Type":"ContainerStarted","Data":"1d34d22d6ebae088d70f97db00d19c736cb78bac2aa56e828979f4433194968b"}
Sep 10 16:00:02.731443 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:02.731413    2564 patch_prober.go:28] interesting pod/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 10 16:00:02.731592 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:02.731471    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7" podUID="fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 10 16:00:02.776950 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:02.776928    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-6bb9m"
Sep 10 16:00:02.915146 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:02.915124    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 16:00:03.057652 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.057565    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istiod-ca-cert\") pod \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") "
Sep 10 16:00:03.057652 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.057601    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-data\") pod \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") "
Sep 10 16:00:03.057652 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.057632    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-token\") pod \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") "
Sep 10 16:00:03.057928 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.057782    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-envoy\") pod \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") "
Sep 10 16:00:03.057928 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.057825    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-podinfo\") pod \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") "
Sep 10 16:00:03.057928 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.057856    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-workload-socket\") pod \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") "
Sep 10 16:00:03.058077 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.057932    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-data" (OuterVolumeSpecName: "istio-data") pod "fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" (UID: "fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:00:03.058077 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.057961    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-workload-certs\") pod \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") "
Sep 10 16:00:03.058077 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.057976    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" (UID: "fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:00:03.058077 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.058002    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-credential-socket\") pod \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") "
Sep 10 16:00:03.058077 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.058055    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57kfs\" (UniqueName: \"kubernetes.io/projected/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-kube-api-access-57kfs\") pod \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\" (UID: \"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b\") "
Sep 10 16:00:03.058334 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.058227    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" (UID: "fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:00:03.058334 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.058245    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" (UID: "fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:00:03.058334 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.058283    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" (UID: "fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:00:03.058533 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.058514    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istiod-ca-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:03.058533 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.058535    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-data\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:03.058688 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.058548    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-workload-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:03.058688 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.058559    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-workload-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:03.058688 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.058575    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-credential-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:03.059962 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.059943    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-token" (OuterVolumeSpecName: "istio-token") pod "fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" (UID: "fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:00:03.060324 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.060305    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-kube-api-access-57kfs" (OuterVolumeSpecName: "kube-api-access-57kfs") pod "fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" (UID: "fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b"). InnerVolumeSpecName "kube-api-access-57kfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:00:03.060511 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.060493    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" (UID: "fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:00:03.060559 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.060497    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" (UID: "fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 16:00:03.159854 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.159823    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57kfs\" (UniqueName: \"kubernetes.io/projected/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-kube-api-access-57kfs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:03.159854 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.159849    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-token\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:03.159854 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.159858    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-envoy\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:03.160077 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.159867    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b-istio-podinfo\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:03.622067 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.621978    2564 generic.go:356] "Generic (PLEG): container finished" podID="6d749003-c1d1-4b55-9e6f-2930a387b05f" containerID="1441ffe072a500cd520ea203aed805636d00325b600ee4fd6904552645200921" exitCode=0
Sep 10 16:00:03.622518 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.622063    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-607b9a-6c7x8" event={"ID":"6d749003-c1d1-4b55-9e6f-2930a387b05f","Type":"ContainerDied","Data":"1441ffe072a500cd520ea203aed805636d00325b600ee4fd6904552645200921"}
Sep 10 16:00:03.623474 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.623459    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-6bb9m"
Sep 10 16:00:03.623548 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.623479    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817600-6bb9m" event={"ID":"34637ef5-80be-4d7b-ad69-cfeacc663e43","Type":"ContainerDied","Data":"26ec1f03ab49275e3dc5102a7e0269fb30a3d5f0bd02d27f619f42c6ec5f854b"}
Sep 10 16:00:03.623548 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.623518    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26ec1f03ab49275e3dc5102a7e0269fb30a3d5f0bd02d27f619f42c6ec5f854b"
Sep 10 16:00:03.625299 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.625278    2564 generic.go:356] "Generic (PLEG): container finished" podID="fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" containerID="8ec3def874b5ee908b09bfd5548756475ca117b5fd003055a4f0c8c4bcd55f97" exitCode=0
Sep 10 16:00:03.625409 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.625336    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7" event={"ID":"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b","Type":"ContainerDied","Data":"8ec3def874b5ee908b09bfd5548756475ca117b5fd003055a4f0c8c4bcd55f97"}
Sep 10 16:00:03.625409 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.625341    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"
Sep 10 16:00:03.625409 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.625363    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7" event={"ID":"fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b","Type":"ContainerDied","Data":"cd6eda0f5a3e82c1dfa543b79ebeea206d9167efe6c6d69b802f96103ed0a7c9"}
Sep 10 16:00:03.625409 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.625378    2564 scope.go:117] "RemoveContainer" containerID="8ec3def874b5ee908b09bfd5548756475ca117b5fd003055a4f0c8c4bcd55f97"
Sep 10 16:00:03.635766 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.635751    2564 scope.go:117] "RemoveContainer" containerID="807a45ea25c10ea8a6e8786fc9db5025bd41696d75a596efd996b1e29faaf4c5"
Sep 10 16:00:03.645246 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.645227    2564 scope.go:117] "RemoveContainer" containerID="8ec3def874b5ee908b09bfd5548756475ca117b5fd003055a4f0c8c4bcd55f97"
Sep 10 16:00:03.645522 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:00:03.645504    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec3def874b5ee908b09bfd5548756475ca117b5fd003055a4f0c8c4bcd55f97\": container with ID starting with 8ec3def874b5ee908b09bfd5548756475ca117b5fd003055a4f0c8c4bcd55f97 not found: ID does not exist" containerID="8ec3def874b5ee908b09bfd5548756475ca117b5fd003055a4f0c8c4bcd55f97"
Sep 10 16:00:03.645599 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.645534    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec3def874b5ee908b09bfd5548756475ca117b5fd003055a4f0c8c4bcd55f97"} err="failed to get container status \"8ec3def874b5ee908b09bfd5548756475ca117b5fd003055a4f0c8c4bcd55f97\": rpc error: code = NotFound desc = could not find container \"8ec3def874b5ee908b09bfd5548756475ca117b5fd003055a4f0c8c4bcd55f97\": container with ID starting with 8ec3def874b5ee908b09bfd5548756475ca117b5fd003055a4f0c8c4bcd55f97 not found: ID does not exist"
Sep 10 16:00:03.645599 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.645563    2564 scope.go:117] "RemoveContainer" containerID="807a45ea25c10ea8a6e8786fc9db5025bd41696d75a596efd996b1e29faaf4c5"
Sep 10 16:00:03.645854 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:00:03.645839    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"807a45ea25c10ea8a6e8786fc9db5025bd41696d75a596efd996b1e29faaf4c5\": container with ID starting with 807a45ea25c10ea8a6e8786fc9db5025bd41696d75a596efd996b1e29faaf4c5 not found: ID does not exist" containerID="807a45ea25c10ea8a6e8786fc9db5025bd41696d75a596efd996b1e29faaf4c5"
Sep 10 16:00:03.645890 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.645860    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"807a45ea25c10ea8a6e8786fc9db5025bd41696d75a596efd996b1e29faaf4c5"} err="failed to get container status \"807a45ea25c10ea8a6e8786fc9db5025bd41696d75a596efd996b1e29faaf4c5\": rpc error: code = NotFound desc = could not find container \"807a45ea25c10ea8a6e8786fc9db5025bd41696d75a596efd996b1e29faaf4c5\": container with ID starting with 807a45ea25c10ea8a6e8786fc9db5025bd41696d75a596efd996b1e29faaf4c5 not found: ID does not exist"
Sep 10 16:00:03.672996 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.672972    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"]
Sep 10 16:00:03.673129 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:03.673004    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9a70cb1a-openshift-default-7f59979748-47mf7"]
Sep 10 16:00:04.773589 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:04.773550    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-607b9a-6c7x8"
Sep 10 16:00:04.875472 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:04.875437    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d749003-c1d1-4b55-9e6f-2930a387b05f-maas-api-service-ca\") pod \"6d749003-c1d1-4b55-9e6f-2930a387b05f\" (UID: \"6d749003-c1d1-4b55-9e6f-2930a387b05f\") "
Sep 10 16:00:04.875796 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:04.875775    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d749003-c1d1-4b55-9e6f-2930a387b05f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d749003-c1d1-4b55-9e6f-2930a387b05f" (UID: "6d749003-c1d1-4b55-9e6f-2930a387b05f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:00:04.976634 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:04.976593    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d749003-c1d1-4b55-9e6f-2930a387b05f-maas-api-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:05.639572 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:05.639541    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-607b9a-6c7x8"
Sep 10 16:00:05.639725 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:05.639578    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-607b9a-6c7x8" event={"ID":"6d749003-c1d1-4b55-9e6f-2930a387b05f","Type":"ContainerDied","Data":"1d34d22d6ebae088d70f97db00d19c736cb78bac2aa56e828979f4433194968b"}
Sep 10 16:00:05.639725 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:05.639654    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d34d22d6ebae088d70f97db00d19c736cb78bac2aa56e828979f4433194968b"
Sep 10 16:00:05.673064 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:05.673032    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" path="/var/lib/kubelet/pods/fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b/volumes"
Sep 10 16:00:06.561692 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.561640    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-678d855bc7-cgqgg" podUID="7311dae7-6fd9-4e55-91b9-0110c33ff8a8" containerName="console" containerID="cri-o://515ff79d2ec53d4d45e9979fff5fa4451f279f5282bbd5afe0cab2498d8d1b02" gracePeriod=15
Sep 10 16:00:06.647178 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.647132    2564 generic.go:356] "Generic (PLEG): container finished" podID="3765fb24-410e-4107-856c-af41b073836b" containerID="2318776fc92194de041ed8b497306e3161c6da995615200c94fce80d7b08d5d0" exitCode=0
Sep 10 16:00:06.647332 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.647198    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg" event={"ID":"3765fb24-410e-4107-856c-af41b073836b","Type":"ContainerDied","Data":"2318776fc92194de041ed8b497306e3161c6da995615200c94fce80d7b08d5d0"}
Sep 10 16:00:06.647332 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.647263    2564 scope.go:117] "RemoveContainer" containerID="f04919c4dfd1a5a3ad150812cc7b6f28cc1f16a01dbbb5ae076410c8f63527c6"
Sep 10 16:00:06.841075 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.841049    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 16:00:06.844245 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.844225    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 16:00:06.996183 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.996126    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-trusted-ca-bundle\") pod \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") "
Sep 10 16:00:06.996383 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.996197    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sskfw\" (UniqueName: \"kubernetes.io/projected/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-kube-api-access-sskfw\") pod \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") "
Sep 10 16:00:06.996383 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.996229    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3765fb24-410e-4107-856c-af41b073836b-istiod-ca-cert\") pod \"3765fb24-410e-4107-856c-af41b073836b\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") "
Sep 10 16:00:06.996383 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.996281    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-service-ca\") pod \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") "
Sep 10 16:00:06.996383 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.996326    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-istio-envoy\") pod \"3765fb24-410e-4107-856c-af41b073836b\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") "
Sep 10 16:00:06.996383 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.996348    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3765fb24-410e-4107-856c-af41b073836b-istio-podinfo\") pod \"3765fb24-410e-4107-856c-af41b073836b\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") "
Sep 10 16:00:06.996647 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.996384    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-console-config\") pod \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") "
Sep 10 16:00:06.996647 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.996413    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3765fb24-410e-4107-856c-af41b073836b-istio-token\") pod \"3765fb24-410e-4107-856c-af41b073836b\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") "
Sep 10 16:00:06.996647 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.996462    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9lr8\" (UniqueName: \"kubernetes.io/projected/3765fb24-410e-4107-856c-af41b073836b-kube-api-access-t9lr8\") pod \"3765fb24-410e-4107-856c-af41b073836b\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") "
Sep 10 16:00:06.996647 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.996494    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-credential-socket\") pod \"3765fb24-410e-4107-856c-af41b073836b\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") "
Sep 10 16:00:06.996647 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.996518    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-istio-data\") pod \"3765fb24-410e-4107-856c-af41b073836b\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") "
Sep 10 16:00:06.996647 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.996554    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-console-serving-cert\") pod \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") "
Sep 10 16:00:06.996647 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.996596    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-workload-socket\") pod \"3765fb24-410e-4107-856c-af41b073836b\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") "
Sep 10 16:00:06.996647 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.996608    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7311dae7-6fd9-4e55-91b9-0110c33ff8a8" (UID: "7311dae7-6fd9-4e55-91b9-0110c33ff8a8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:00:06.996647 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.996625    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-oauth-serving-cert\") pod \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") "
Sep 10 16:00:06.997065 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.996655    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-workload-certs\") pod \"3765fb24-410e-4107-856c-af41b073836b\" (UID: \"3765fb24-410e-4107-856c-af41b073836b\") "
Sep 10 16:00:06.997065 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.996700    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-console-oauth-config\") pod \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\" (UID: \"7311dae7-6fd9-4e55-91b9-0110c33ff8a8\") "
Sep 10 16:00:06.997065 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.996919    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3765fb24-410e-4107-856c-af41b073836b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3765fb24-410e-4107-856c-af41b073836b" (UID: "3765fb24-410e-4107-856c-af41b073836b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:00:06.997065 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.996917    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3765fb24-410e-4107-856c-af41b073836b" (UID: "3765fb24-410e-4107-856c-af41b073836b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:00:06.997065 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.996970    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-service-ca" (OuterVolumeSpecName: "service-ca") pod "7311dae7-6fd9-4e55-91b9-0110c33ff8a8" (UID: "7311dae7-6fd9-4e55-91b9-0110c33ff8a8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:00:06.997065 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.997060    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:06.997420 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.997077    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-credential-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:06.997420 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.997092    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-trusted-ca-bundle\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:06.997420 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.997107    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3765fb24-410e-4107-856c-af41b073836b-istiod-ca-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:06.997420 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.997336    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-istio-data" (OuterVolumeSpecName: "istio-data") pod "3765fb24-410e-4107-856c-af41b073836b" (UID: "3765fb24-410e-4107-856c-af41b073836b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:00:06.998461 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.998380    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3765fb24-410e-4107-856c-af41b073836b" (UID: "3765fb24-410e-4107-856c-af41b073836b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:00:06.998461 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.998415    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3765fb24-410e-4107-856c-af41b073836b" (UID: "3765fb24-410e-4107-856c-af41b073836b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:00:06.998461 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.998439    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7311dae7-6fd9-4e55-91b9-0110c33ff8a8" (UID: "7311dae7-6fd9-4e55-91b9-0110c33ff8a8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:00:06.998711 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.998596    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-console-config" (OuterVolumeSpecName: "console-config") pod "7311dae7-6fd9-4e55-91b9-0110c33ff8a8" (UID: "7311dae7-6fd9-4e55-91b9-0110c33ff8a8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:00:06.999983 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:06.999954    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3765fb24-410e-4107-856c-af41b073836b-kube-api-access-t9lr8" (OuterVolumeSpecName: "kube-api-access-t9lr8") pod "3765fb24-410e-4107-856c-af41b073836b" (UID: "3765fb24-410e-4107-856c-af41b073836b"). InnerVolumeSpecName "kube-api-access-t9lr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:00:07.001188 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.001126    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3765fb24-410e-4107-856c-af41b073836b" (UID: "3765fb24-410e-4107-856c-af41b073836b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:00:07.001346 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.001309    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7311dae7-6fd9-4e55-91b9-0110c33ff8a8" (UID: "7311dae7-6fd9-4e55-91b9-0110c33ff8a8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:00:07.001466 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.001432    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3765fb24-410e-4107-856c-af41b073836b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3765fb24-410e-4107-856c-af41b073836b" (UID: "3765fb24-410e-4107-856c-af41b073836b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 16:00:07.001596 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.001572    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7311dae7-6fd9-4e55-91b9-0110c33ff8a8" (UID: "7311dae7-6fd9-4e55-91b9-0110c33ff8a8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:00:07.001742 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.001725    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3765fb24-410e-4107-856c-af41b073836b-istio-token" (OuterVolumeSpecName: "istio-token") pod "3765fb24-410e-4107-856c-af41b073836b" (UID: "3765fb24-410e-4107-856c-af41b073836b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:00:07.002062 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.002042    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-kube-api-access-sskfw" (OuterVolumeSpecName: "kube-api-access-sskfw") pod "7311dae7-6fd9-4e55-91b9-0110c33ff8a8" (UID: "7311dae7-6fd9-4e55-91b9-0110c33ff8a8"). InnerVolumeSpecName "kube-api-access-sskfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:00:07.098546 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.098469    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-console-serving-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:07.098546 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.098494    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-workload-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:07.098546 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.098508    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-oauth-serving-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:07.098546 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.098521    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-workload-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:07.098546 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.098532    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-console-oauth-config\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:07.098546 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.098540    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sskfw\" (UniqueName: \"kubernetes.io/projected/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-kube-api-access-sskfw\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:07.098546 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.098549    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-istio-envoy\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:07.098864 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.098557    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3765fb24-410e-4107-856c-af41b073836b-istio-podinfo\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:07.098864 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.098565    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7311dae7-6fd9-4e55-91b9-0110c33ff8a8-console-config\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:07.098864 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.098573    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3765fb24-410e-4107-856c-af41b073836b-istio-token\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:07.098864 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.098580    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9lr8\" (UniqueName: \"kubernetes.io/projected/3765fb24-410e-4107-856c-af41b073836b-kube-api-access-t9lr8\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:07.098864 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.098587    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3765fb24-410e-4107-856c-af41b073836b-istio-data\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:00:07.655005 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.654966    2564 generic.go:356] "Generic (PLEG): container finished" podID="7311dae7-6fd9-4e55-91b9-0110c33ff8a8" containerID="515ff79d2ec53d4d45e9979fff5fa4451f279f5282bbd5afe0cab2498d8d1b02" exitCode=0
Sep 10 16:00:07.655473 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.655017    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678d855bc7-cgqgg"
Sep 10 16:00:07.655473 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.655055    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678d855bc7-cgqgg" event={"ID":"7311dae7-6fd9-4e55-91b9-0110c33ff8a8","Type":"ContainerDied","Data":"515ff79d2ec53d4d45e9979fff5fa4451f279f5282bbd5afe0cab2498d8d1b02"}
Sep 10 16:00:07.655473 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.655106    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678d855bc7-cgqgg" event={"ID":"7311dae7-6fd9-4e55-91b9-0110c33ff8a8","Type":"ContainerDied","Data":"98234071a5711512b6c6d05f97b33f3eebf013b1ed467eb7c75ad59f3018a307"}
Sep 10 16:00:07.655473 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.655128    2564 scope.go:117] "RemoveContainer" containerID="515ff79d2ec53d4d45e9979fff5fa4451f279f5282bbd5afe0cab2498d8d1b02"
Sep 10 16:00:07.656721 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.656670    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"
Sep 10 16:00:07.656721 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.656714    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg" event={"ID":"3765fb24-410e-4107-856c-af41b073836b","Type":"ContainerDied","Data":"26644aa2b71e2341108cd7f5e93fe1d66d6d0e3a1163eb909fc4d7ae9c3bbd2b"}
Sep 10 16:00:07.666122 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.666103    2564 scope.go:117] "RemoveContainer" containerID="515ff79d2ec53d4d45e9979fff5fa4451f279f5282bbd5afe0cab2498d8d1b02"
Sep 10 16:00:07.666510 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:00:07.666489    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515ff79d2ec53d4d45e9979fff5fa4451f279f5282bbd5afe0cab2498d8d1b02\": container with ID starting with 515ff79d2ec53d4d45e9979fff5fa4451f279f5282bbd5afe0cab2498d8d1b02 not found: ID does not exist" containerID="515ff79d2ec53d4d45e9979fff5fa4451f279f5282bbd5afe0cab2498d8d1b02"
Sep 10 16:00:07.666605 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.666516    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515ff79d2ec53d4d45e9979fff5fa4451f279f5282bbd5afe0cab2498d8d1b02"} err="failed to get container status \"515ff79d2ec53d4d45e9979fff5fa4451f279f5282bbd5afe0cab2498d8d1b02\": rpc error: code = NotFound desc = could not find container \"515ff79d2ec53d4d45e9979fff5fa4451f279f5282bbd5afe0cab2498d8d1b02\": container with ID starting with 515ff79d2ec53d4d45e9979fff5fa4451f279f5282bbd5afe0cab2498d8d1b02 not found: ID does not exist"
Sep 10 16:00:07.666605 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.666533    2564 scope.go:117] "RemoveContainer" containerID="2318776fc92194de041ed8b497306e3161c6da995615200c94fce80d7b08d5d0"
Sep 10 16:00:07.683458 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.683436    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-678d855bc7-cgqgg"]
Sep 10 16:00:07.688318 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.688285    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-678d855bc7-cgqgg"]
Sep 10 16:00:07.700254 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.700230    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"]
Sep 10 16:00:07.709301 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:07.709280    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-83dcaf-openshift-default-85dd9469dd-9qhkg"]
Sep 10 16:00:09.675227 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:09.675188    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3765fb24-410e-4107-856c-af41b073836b" path="/var/lib/kubelet/pods/3765fb24-410e-4107-856c-af41b073836b/volumes"
Sep 10 16:00:09.675757 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:09.675741    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7311dae7-6fd9-4e55-91b9-0110c33ff8a8" path="/var/lib/kubelet/pods/7311dae7-6fd9-4e55-91b9-0110c33ff8a8/volumes"
Sep 10 16:00:12.747240 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:12.747211    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-607b9a-6c7x8"]
Sep 10 16:00:12.756377 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:12.756350    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-607b9a-6c7x8"]
Sep 10 16:00:13.673183 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.673120    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d749003-c1d1-4b55-9e6f-2930a387b05f" path="/var/lib/kubelet/pods/6d749003-c1d1-4b55-9e6f-2930a387b05f/volumes"
Sep 10 16:00:13.865118 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.865087    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"]
Sep 10 16:00:13.865542 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.865528    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d749003-c1d1-4b55-9e6f-2930a387b05f" containerName="revoke-keys"
Sep 10 16:00:13.865600 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.865544    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d749003-c1d1-4b55-9e6f-2930a387b05f" containerName="revoke-keys"
Sep 10 16:00:13.865600 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.865565    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" containerName="istio-proxy"
Sep 10 16:00:13.865600 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.865572    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" containerName="istio-proxy"
Sep 10 16:00:13.865600 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.865590    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3765fb24-410e-4107-856c-af41b073836b" containerName="istio-proxy"
Sep 10 16:00:13.865600 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.865595    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3765fb24-410e-4107-856c-af41b073836b" containerName="istio-proxy"
Sep 10 16:00:13.865759 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.865603    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34637ef5-80be-4d7b-ad69-cfeacc663e43" containerName="cleanup"
Sep 10 16:00:13.865759 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.865610    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="34637ef5-80be-4d7b-ad69-cfeacc663e43" containerName="cleanup"
Sep 10 16:00:13.865759 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.865624    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3765fb24-410e-4107-856c-af41b073836b" containerName="istio-proxy"
Sep 10 16:00:13.865759 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.865630    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3765fb24-410e-4107-856c-af41b073836b" containerName="istio-proxy"
Sep 10 16:00:13.865759 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.865637    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7311dae7-6fd9-4e55-91b9-0110c33ff8a8" containerName="console"
Sep 10 16:00:13.865759 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.865643    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7311dae7-6fd9-4e55-91b9-0110c33ff8a8" containerName="console"
Sep 10 16:00:13.865759 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.865699    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" containerName="istio-proxy"
Sep 10 16:00:13.865759 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.865708    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3765fb24-410e-4107-856c-af41b073836b" containerName="istio-proxy"
Sep 10 16:00:13.865759 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.865713    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d749003-c1d1-4b55-9e6f-2930a387b05f" containerName="revoke-keys"
Sep 10 16:00:13.865759 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.865720    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="34637ef5-80be-4d7b-ad69-cfeacc663e43" containerName="cleanup"
Sep 10 16:00:13.865759 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.865727    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" containerName="istio-proxy"
Sep 10 16:00:13.865759 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.865732    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7311dae7-6fd9-4e55-91b9-0110c33ff8a8" containerName="console"
Sep 10 16:00:13.865759 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.865738    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3765fb24-410e-4107-856c-af41b073836b" containerName="istio-proxy"
Sep 10 16:00:13.870173 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.870139    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:13.872586 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.872567    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-247f70-openshift-default-dockercfg-7n6f2\""
Sep 10 16:00:13.882109 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.882086    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"]
Sep 10 16:00:13.974552 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.974455    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-data\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:13.974552 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.974499    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-workload-socket\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:13.974795 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.974552    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-workload-certs\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:13.974795 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.974599    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-token\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:13.974795 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.974662    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e66617-3de1-49ce-9d8b-0bc626c24765-istiod-ca-cert\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:13.974795 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.974688    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx95h\" (UniqueName: \"kubernetes.io/projected/45e66617-3de1-49ce-9d8b-0bc626c24765-kube-api-access-mx95h\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:13.974795 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.974712    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-envoy\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:13.975067 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.974814    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-credential-socket\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:13.975067 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:13.974850    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-podinfo\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:14.076330 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.076289    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-workload-certs\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:14.076330 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.076338    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-token\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:14.076549 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.076470    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e66617-3de1-49ce-9d8b-0bc626c24765-istiod-ca-cert\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:14.076586 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.076547    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx95h\" (UniqueName: \"kubernetes.io/projected/45e66617-3de1-49ce-9d8b-0bc626c24765-kube-api-access-mx95h\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:14.076586 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.076578    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-envoy\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:14.076664 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.076647    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-credential-socket\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:14.076709 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.076695    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-podinfo\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:14.076757 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.076739    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-workload-certs\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:14.076811 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.076774    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-data\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:14.076870 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.076810    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-workload-socket\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:14.077027 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.077001    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-credential-socket\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:14.077121 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.077100    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-workload-socket\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:14.077209 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.077143    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-data\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:14.077209 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.077142    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e66617-3de1-49ce-9d8b-0bc626c24765-istiod-ca-cert\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:14.079148 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.079125    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-envoy\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:14.079396 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.079373    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-podinfo\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:14.085347 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.085322    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-token\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:14.085506 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.085488    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx95h\" (UniqueName: \"kubernetes.io/projected/45e66617-3de1-49ce-9d8b-0bc626c24765-kube-api-access-mx95h\") pod \"e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") " pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:14.181261 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.181230    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:14.464364 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.464337    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"]
Sep 10 16:00:14.466415 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:00:14.466389    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e66617_3de1_49ce_9d8b_0bc626c24765.slice/crio-c796c2cbb22f5cb3b0ff00ff89b6969cb797c77e182acca995654c999f4ef32c WatchSource:0}: Error finding container c796c2cbb22f5cb3b0ff00ff89b6969cb797c77e182acca995654c999f4ef32c: Status 404 returned error can't find the container with id c796c2cbb22f5cb3b0ff00ff89b6969cb797c77e182acca995654c999f4ef32c
Sep 10 16:00:14.468803 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.468769    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:00:14.468880 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.468833    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:00:14.468880 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.468864    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:00:14.701371 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.701322    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x" event={"ID":"45e66617-3de1-49ce-9d8b-0bc626c24765","Type":"ContainerStarted","Data":"79a5e3ea3873e7721a42f5c585e890a76e3c06b0585ae9855021b597550c29ad"}
Sep 10 16:00:14.701545 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.701400    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x" event={"ID":"45e66617-3de1-49ce-9d8b-0bc626c24765","Type":"ContainerStarted","Data":"c796c2cbb22f5cb3b0ff00ff89b6969cb797c77e182acca995654c999f4ef32c"}
Sep 10 16:00:14.726275 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:14.726214    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x" podStartSLOduration=1.7261914630000001 podStartE2EDuration="1.726191463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:14.722972645 +0000 UTC m=+1093.527291911" watchObservedRunningTime="2026-09-10 16:00:14.726191463 +0000 UTC m=+1093.530510721"
Sep 10 16:00:15.181553 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:15.181514    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:15.187448 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:15.187392    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:15.707076 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:15.707039    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:15.707973 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:15.707952    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:21.544629 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.544590    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"]
Sep 10 16:00:21.545516 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.545480    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" containerName="istio-proxy"
Sep 10 16:00:21.545516 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.545506    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb61d2eb-8d3f-49e9-8d12-8a34abbbb93b" containerName="istio-proxy"
Sep 10 16:00:21.548431 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.548410    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.551514 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.551490    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-b52270-openshift-default-dockercfg-fg7dv\""
Sep 10 16:00:21.565859 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.565837    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"]
Sep 10 16:00:21.655219 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.655145    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-podinfo\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.655219 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.655215    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-data\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.655496 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.655328    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-credential-socket\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.655496 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.655395    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-workload-socket\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.655496 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.655420    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-workload-certs\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.655496 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.655445    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istiod-ca-cert\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.655718 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.655507    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-envoy\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.655718 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.655531    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntrvz\" (UniqueName: \"kubernetes.io/projected/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-kube-api-access-ntrvz\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.655718 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.655568    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-token\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.756368 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.756331    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-podinfo\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.756368 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.756370    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-data\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.756624 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.756405    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-credential-socket\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.756624 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.756438    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-workload-socket\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.756624 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.756456    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-workload-certs\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.756624 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.756471    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istiod-ca-cert\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.756624 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.756488    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-envoy\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.756624 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.756508    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntrvz\" (UniqueName: \"kubernetes.io/projected/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-kube-api-access-ntrvz\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.756624 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.756553    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-token\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.756977 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.756813    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-data\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.756977 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.756863    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-workload-socket\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.756977 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.756945    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-workload-certs\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.757101 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.756977    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-credential-socket\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.757250 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.757230    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istiod-ca-cert\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.759071 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.759053    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-envoy\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.759202 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.759181    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-podinfo\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.766266 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.766242    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-token\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.766812 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.766794    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntrvz\" (UniqueName: \"kubernetes.io/projected/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-kube-api-access-ntrvz\") pod \"e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") " pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:21.861304 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:21.861207    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:22.145774 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:22.145743    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"]
Sep 10 16:00:22.147648 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:00:22.147619    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe09b0b_462a_4b7a_a95b_8ed91494a7e8.slice/crio-077fa8f85e1d9fa75b003b0fa394fcb2c132dc5192c3e308caf32b16e9d37a6a WatchSource:0}: Error finding container 077fa8f85e1d9fa75b003b0fa394fcb2c132dc5192c3e308caf32b16e9d37a6a: Status 404 returned error can't find the container with id 077fa8f85e1d9fa75b003b0fa394fcb2c132dc5192c3e308caf32b16e9d37a6a
Sep 10 16:00:22.150038 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:22.149998    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:00:22.150149 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:22.150095    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:00:22.150289 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:22.150145    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:00:22.750081 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:22.750042    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6" event={"ID":"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8","Type":"ContainerStarted","Data":"c0db2a278ab1ec8b3ece0e2b3f407f4902107354689d110a9f33477f8b907a0b"}
Sep 10 16:00:22.750486 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:22.750102    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6" event={"ID":"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8","Type":"ContainerStarted","Data":"077fa8f85e1d9fa75b003b0fa394fcb2c132dc5192c3e308caf32b16e9d37a6a"}
Sep 10 16:00:22.773309 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:22.773264    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6" podStartSLOduration=1.773250554 podStartE2EDuration="1.773250554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:22.77313352 +0000 UTC m=+1101.577452782" watchObservedRunningTime="2026-09-10 16:00:22.773250554 +0000 UTC m=+1101.577569815"
Sep 10 16:00:22.862265 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:22.862220    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:23.867563 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:23.867534    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:24.768844 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:24.768799    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:24.769961 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:24.769934    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:30.714724 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:30.714678    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": read tcp 10.133.0.2:50890->10.133.0.73:15021: read: connection reset by peer" start-of-body=
Sep 10 16:00:30.715108 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:30.714756    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x" podUID="45e66617-3de1-49ce-9d8b-0bc626c24765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": read tcp 10.133.0.2:50890->10.133.0.73:15021: read: connection reset by peer"
Sep 10 16:00:30.808140 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:30.808109    2564 generic.go:356] "Generic (PLEG): container finished" podID="45e66617-3de1-49ce-9d8b-0bc626c24765" containerID="79a5e3ea3873e7721a42f5c585e890a76e3c06b0585ae9855021b597550c29ad" exitCode=0
Sep 10 16:00:30.808318 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:30.808199    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x" event={"ID":"45e66617-3de1-49ce-9d8b-0bc626c24765","Type":"ContainerDied","Data":"79a5e3ea3873e7721a42f5c585e890a76e3c06b0585ae9855021b597550c29ad"}
Sep 10 16:00:30.808653 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:30.808634    2564 scope.go:117] "RemoveContainer" containerID="79a5e3ea3873e7721a42f5c585e890a76e3c06b0585ae9855021b597550c29ad"
Sep 10 16:00:30.809750 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:30.809719    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:00:30.809856 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:30.809787    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:00:30.809856 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:30.809818    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:00:31.822046 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:31.822005    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x" event={"ID":"45e66617-3de1-49ce-9d8b-0bc626c24765","Type":"ContainerStarted","Data":"62a60136deb007512d06b421ceb8f7a922fbf1c71cb0753e1b962c53b4d47e70"}
Sep 10 16:00:32.182405 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:32.182365    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:32.182653 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:32.182629    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 10 16:00:32.182742 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:32.182686    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x" podUID="45e66617-3de1-49ce-9d8b-0bc626c24765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 10 16:00:33.181766 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:33.181733    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 10 16:00:33.182191 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:33.181815    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x" podUID="45e66617-3de1-49ce-9d8b-0bc626c24765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 10 16:00:34.182533 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:34.182497    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 10 16:00:34.182935 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:34.182563    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x" podUID="45e66617-3de1-49ce-9d8b-0bc626c24765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 10 16:00:35.182625 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:35.182583    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 10 16:00:35.183008 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:35.182652    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x" podUID="45e66617-3de1-49ce-9d8b-0bc626c24765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 10 16:00:36.182527 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:36.182488    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 10 16:00:36.182715 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:36.182545    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x" podUID="45e66617-3de1-49ce-9d8b-0bc626c24765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 10 16:00:37.185251 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:37.185215    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:37.185729 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:37.185514    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:37.186205 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:37.186190    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:00:40.810388 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:40.810346    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 16:00:40.810916 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:40.810457    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6" podUID="6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 16:00:42.304085 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.304049    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"]
Sep 10 16:00:42.308533 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.308507    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.311235 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.311208    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-298d9b-openshift-default-dockercfg-4sbz2\""
Sep 10 16:00:42.332195 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.332137    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"]
Sep 10 16:00:42.386659 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.386619    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-podinfo\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.386659 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.386679    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-workload-certs\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.386659 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.386724    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-envoy\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.386659 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.386802    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istiod-ca-cert\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.386659 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.386848    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-credential-socket\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.386659 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.386878    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-data\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.386659 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.386950    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-token\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.386659 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.387009    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-workload-socket\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.386659 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.387052    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lds47\" (UniqueName: \"kubernetes.io/projected/4ac0d739-9287-4fbf-a747-2ed9ae69f969-kube-api-access-lds47\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.488624 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.488575    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istiod-ca-cert\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.488854 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.488656    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-credential-socket\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.488854 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.488691    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-data\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.488854 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.488742    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-token\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.488854 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.488773    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-workload-socket\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.488854 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.488807    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lds47\" (UniqueName: \"kubernetes.io/projected/4ac0d739-9287-4fbf-a747-2ed9ae69f969-kube-api-access-lds47\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.489201 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.488879    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-podinfo\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.489201 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.488918    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-workload-certs\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.489201 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.488945    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-envoy\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.489397 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.489373    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istiod-ca-cert\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.489693 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.489666    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-credential-socket\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.489813 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.489719    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-workload-socket\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.489813 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.489756    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-workload-certs\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.490218 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.490194    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-data\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.492394 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.492367    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-envoy\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.492667 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.492645    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-podinfo\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.502371 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.502343    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lds47\" (UniqueName: \"kubernetes.io/projected/4ac0d739-9287-4fbf-a747-2ed9ae69f969-kube-api-access-lds47\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.502778 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.502753    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-token\") pod \"e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") " pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.622471 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.622374    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:42.909553 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.909468    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"]
Sep 10 16:00:42.912944 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:00:42.912917    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac0d739_9287_4fbf_a747_2ed9ae69f969.slice/crio-7858f6880bf2cb66f384c179fff3a19dc72532275f660a3e2c6f3caa0bd2285e WatchSource:0}: Error finding container 7858f6880bf2cb66f384c179fff3a19dc72532275f660a3e2c6f3caa0bd2285e: Status 404 returned error can't find the container with id 7858f6880bf2cb66f384c179fff3a19dc72532275f660a3e2c6f3caa0bd2285e
Sep 10 16:00:42.915509 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.915468    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:00:42.915580 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.915567    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:00:42.915632 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:42.915616    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:00:43.920586 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:43.920547    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt" event={"ID":"4ac0d739-9287-4fbf-a747-2ed9ae69f969","Type":"ContainerStarted","Data":"324c47b2c289e9e8daf82a71f0bbcfdec325aa1e965e1ef459756d1647b2b3d1"}
Sep 10 16:00:43.921033 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:43.920604    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt" event={"ID":"4ac0d739-9287-4fbf-a747-2ed9ae69f969","Type":"ContainerStarted","Data":"7858f6880bf2cb66f384c179fff3a19dc72532275f660a3e2c6f3caa0bd2285e"}
Sep 10 16:00:43.922207 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:43.922186    2564 generic.go:356] "Generic (PLEG): container finished" podID="6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" containerID="c0db2a278ab1ec8b3ece0e2b3f407f4902107354689d110a9f33477f8b907a0b" exitCode=0
Sep 10 16:00:43.922334 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:43.922254    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6" event={"ID":"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8","Type":"ContainerDied","Data":"c0db2a278ab1ec8b3ece0e2b3f407f4902107354689d110a9f33477f8b907a0b"}
Sep 10 16:00:43.922638 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:43.922625    2564 scope.go:117] "RemoveContainer" containerID="c0db2a278ab1ec8b3ece0e2b3f407f4902107354689d110a9f33477f8b907a0b"
Sep 10 16:00:43.923699 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:43.923659    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:00:43.923792 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:43.923751    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:00:43.923792 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:43.923784    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:00:43.944473 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:43.944431    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt" podStartSLOduration=1.944416334 podStartE2EDuration="1.944416334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:43.943146378 +0000 UTC m=+1122.747465639" watchObservedRunningTime="2026-09-10 16:00:43.944416334 +0000 UTC m=+1122.748735596"
Sep 10 16:00:44.624260 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:44.624216    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:44.629503 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:44.629471    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:44.932140 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:44.932103    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6" event={"ID":"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8","Type":"ContainerStarted","Data":"409769f954d4370ace4b83feb561472d0dd2003afdac323e15ade8525bc13dd3"}
Sep 10 16:00:44.932650 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:44.932538    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:44.933622 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:44.933604    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:00:45.862084 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:45.862051    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:45.862328 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:45.862305    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 10 16:00:45.862387 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:45.862366    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6" podUID="6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 10 16:00:46.862021 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:46.861987    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 10 16:00:46.862446 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:46.862057    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6" podUID="6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 10 16:00:47.862917 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:47.862456    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 10 16:00:47.862917 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:47.862545    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6" podUID="6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 10 16:00:48.862284 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:48.862240    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 10 16:00:48.862471 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:48.862315    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6" podUID="6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 10 16:00:49.862462 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:49.862417    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 10 16:00:49.862941 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:49.862483    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6" podUID="6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 10 16:00:50.865282 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:50.865251    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:50.865745 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:50.865636    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:50.866361 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:50.866344    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:00:55.474453 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.474423    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"]
Sep 10 16:00:55.477209 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.477187    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.479798 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.479769    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-108dbb-openshift-default-dockercfg-cdqnr\""
Sep 10 16:00:55.489342 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.489319    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"]
Sep 10 16:00:55.535152 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.535122    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hndgn\" (UniqueName: \"kubernetes.io/projected/f413d5cd-e143-4982-8e07-ede979564650-kube-api-access-hndgn\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.535360 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.535239    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f413d5cd-e143-4982-8e07-ede979564650-istio-token\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.535360 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.535321    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-istio-envoy\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.535360 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.535358    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-workload-certs\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.535517 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.535381    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f413d5cd-e143-4982-8e07-ede979564650-istiod-ca-cert\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.535517 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.535451    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f413d5cd-e143-4982-8e07-ede979564650-istio-podinfo\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.535517 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.535481    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-workload-socket\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.535668 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.535533    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-credential-socket\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.535668 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.535601    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-istio-data\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.637033 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.636991    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hndgn\" (UniqueName: \"kubernetes.io/projected/f413d5cd-e143-4982-8e07-ede979564650-kube-api-access-hndgn\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.637245 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.637100    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f413d5cd-e143-4982-8e07-ede979564650-istio-token\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.637245 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.637178    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-istio-envoy\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.637245 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.637208    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-workload-certs\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.637245 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.637236    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f413d5cd-e143-4982-8e07-ede979564650-istiod-ca-cert\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.637480 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.637273    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f413d5cd-e143-4982-8e07-ede979564650-istio-podinfo\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.637480 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.637308    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-workload-socket\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.637480 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.637352    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-credential-socket\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.637480 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.637386    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-istio-data\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.637823 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.637800    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-istio-data\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.638003 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.637852    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-workload-certs\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.638174 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.638135    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-credential-socket\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.638255 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.638217    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f413d5cd-e143-4982-8e07-ede979564650-istiod-ca-cert\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.638399 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.638382    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-workload-socket\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.640148 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.640075    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f413d5cd-e143-4982-8e07-ede979564650-istio-podinfo\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.640415 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.640395    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-istio-envoy\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.645863 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.645843    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hndgn\" (UniqueName: \"kubernetes.io/projected/f413d5cd-e143-4982-8e07-ede979564650-kube-api-access-hndgn\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.646058 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.646038    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f413d5cd-e143-4982-8e07-ede979564650-istio-token\") pod \"e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") " pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:55.789321 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:55.789221    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:56.090915 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:56.090838    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"]
Sep 10 16:00:56.094312 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:00:56.094281    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf413d5cd_e143_4982_8e07_ede979564650.slice/crio-ca406d671ca7869d489148d32896f82eeda3f7093f6a1f2c5f60a88370602462 WatchSource:0}: Error finding container ca406d671ca7869d489148d32896f82eeda3f7093f6a1f2c5f60a88370602462: Status 404 returned error can't find the container with id ca406d671ca7869d489148d32896f82eeda3f7093f6a1f2c5f60a88370602462
Sep 10 16:00:56.099414 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:56.097056    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:00:56.099414 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:56.097176    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:00:56.099414 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:56.097224    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:00:57.029936 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:57.029895    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw" event={"ID":"f413d5cd-e143-4982-8e07-ede979564650","Type":"ContainerStarted","Data":"7fdbe5def722097ddc2342a15f3bdf23d9e9f72c53f425d502353ccbe8f98540"}
Sep 10 16:00:57.030425 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:57.029955    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw" event={"ID":"f413d5cd-e143-4982-8e07-ede979564650","Type":"ContainerStarted","Data":"ca406d671ca7869d489148d32896f82eeda3f7093f6a1f2c5f60a88370602462"}
Sep 10 16:00:57.057871 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:57.057822    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw" podStartSLOduration=2.0578084 podStartE2EDuration="2.0578084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:57.055267492 +0000 UTC m=+1135.859586778" watchObservedRunningTime="2026-09-10 16:00:57.0578084 +0000 UTC m=+1135.862127661"
Sep 10 16:00:57.789877 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:57.789848    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:57.794909 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:57.794882    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:58.036229 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:58.036197    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:00:58.037240 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:00:58.037218    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:01:04.078849 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:04.078817    2564 generic.go:356] "Generic (PLEG): container finished" podID="4ac0d739-9287-4fbf-a747-2ed9ae69f969" containerID="324c47b2c289e9e8daf82a71f0bbcfdec325aa1e965e1ef459756d1647b2b3d1" exitCode=0
Sep 10 16:01:04.079278 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:04.078887    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt" event={"ID":"4ac0d739-9287-4fbf-a747-2ed9ae69f969","Type":"ContainerDied","Data":"324c47b2c289e9e8daf82a71f0bbcfdec325aa1e965e1ef459756d1647b2b3d1"}
Sep 10 16:01:04.079403 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:04.079385    2564 scope.go:117] "RemoveContainer" containerID="324c47b2c289e9e8daf82a71f0bbcfdec325aa1e965e1ef459756d1647b2b3d1"
Sep 10 16:01:04.080729 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:04.080694    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:01:04.080798 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:04.080776    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:01:04.080844 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:04.080810    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:01:05.092444 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:05.092404    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt" event={"ID":"4ac0d739-9287-4fbf-a747-2ed9ae69f969","Type":"ContainerStarted","Data":"2760a249aa20e7ffd4a88d7958480de7d9e15dfeffb72df1e955d527194986b4"}
Sep 10 16:01:05.623436 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:05.623399    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:01:05.623662 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:05.623635    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 10 16:01:05.623747 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:05.623698    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt" podUID="4ac0d739-9287-4fbf-a747-2ed9ae69f969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 10 16:01:06.623174 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:06.623105    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 10 16:01:06.623684 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:06.623205    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt" podUID="4ac0d739-9287-4fbf-a747-2ed9ae69f969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 10 16:01:07.623700 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:07.623653    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 10 16:01:07.624246 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:07.623728    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt" podUID="4ac0d739-9287-4fbf-a747-2ed9ae69f969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 10 16:01:08.623148 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:08.623109    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 10 16:01:08.623497 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:08.623459    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt" podUID="4ac0d739-9287-4fbf-a747-2ed9ae69f969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 10 16:01:09.623059 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:09.623017    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 10 16:01:09.623582 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:09.623097    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt" podUID="4ac0d739-9287-4fbf-a747-2ed9ae69f969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 10 16:01:10.627611 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:10.627581    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:01:10.628058 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:10.627871    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:01:10.628742 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:10.628717    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:01:20.206384 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.206351    2564 generic.go:356] "Generic (PLEG): container finished" podID="f413d5cd-e143-4982-8e07-ede979564650" containerID="7fdbe5def722097ddc2342a15f3bdf23d9e9f72c53f425d502353ccbe8f98540" exitCode=0
Sep 10 16:01:20.206975 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.206412    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw" event={"ID":"f413d5cd-e143-4982-8e07-ede979564650","Type":"ContainerDied","Data":"7fdbe5def722097ddc2342a15f3bdf23d9e9f72c53f425d502353ccbe8f98540"}
Sep 10 16:01:20.206975 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.206876    2564 scope.go:117] "RemoveContainer" containerID="7fdbe5def722097ddc2342a15f3bdf23d9e9f72c53f425d502353ccbe8f98540"
Sep 10 16:01:20.208339 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.208300    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:01:20.208471 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.208390    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:01:20.208471 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.208436    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:01:20.233292 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.233263    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"]
Sep 10 16:01:20.237957 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.237935    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.240810 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.240793    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-69972456-openshift-default-dockercfg-zwb8s\""
Sep 10 16:01:20.255898 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.255868    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"]
Sep 10 16:01:20.311009 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.310969    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-workload-socket\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.311210 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.311075    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-credential-socket\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.311210 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.311132    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74161930-5277-4a96-a3ab-c263334800b2-istio-token\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.311354 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.311295    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74161930-5277-4a96-a3ab-c263334800b2-istio-podinfo\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.311412 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.311362    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-workload-certs\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.311485 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.311465    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-istio-data\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.311543 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.311497    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74161930-5277-4a96-a3ab-c263334800b2-istiod-ca-cert\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.311599 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.311563    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7shr\" (UniqueName: \"kubernetes.io/projected/74161930-5277-4a96-a3ab-c263334800b2-kube-api-access-p7shr\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.311650 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.311603    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-istio-envoy\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.412734 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.412699    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-istio-data\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.412734 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.412738    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74161930-5277-4a96-a3ab-c263334800b2-istiod-ca-cert\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.412987 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.412768    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7shr\" (UniqueName: \"kubernetes.io/projected/74161930-5277-4a96-a3ab-c263334800b2-kube-api-access-p7shr\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.412987 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.412792    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-istio-envoy\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.412987 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.412833    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-workload-socket\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.412987 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.412870    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-credential-socket\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.412987 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.412914    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74161930-5277-4a96-a3ab-c263334800b2-istio-token\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.412987 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.412959    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74161930-5277-4a96-a3ab-c263334800b2-istio-podinfo\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.412987 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.412981    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-workload-certs\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.413390 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.413207    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-istio-data\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.413390 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.413256    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-workload-socket\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.413390 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.413324    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-workload-certs\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.413578 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.413507    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-credential-socket\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.413985 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.413958    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74161930-5277-4a96-a3ab-c263334800b2-istiod-ca-cert\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.415306 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.415286    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-istio-envoy\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.415768 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.415750    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74161930-5277-4a96-a3ab-c263334800b2-istio-podinfo\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.421930 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.421902    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74161930-5277-4a96-a3ab-c263334800b2-istio-token\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.422398 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.422379    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7shr\" (UniqueName: \"kubernetes.io/projected/74161930-5277-4a96-a3ab-c263334800b2-kube-api-access-p7shr\") pod \"e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") " pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.562020 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.561925    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:20.851460 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.851426    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"]
Sep 10 16:01:20.860091 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.860027    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:01:20.860246 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.860187    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:01:20.860318 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:20.860244    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:01:21.221019 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:21.219966    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw" event={"ID":"f413d5cd-e143-4982-8e07-ede979564650","Type":"ContainerStarted","Data":"67cf9c48a3294a238e76f09eaec91c3259ace60af98b802cdf524e6ddd6de644"}
Sep 10 16:01:21.231198 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:21.230447    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh" event={"ID":"74161930-5277-4a96-a3ab-c263334800b2","Type":"ContainerStarted","Data":"c9f112201c8224938d8b425c4fdd486a01bd8950ee51457663f7e2f733930c02"}
Sep 10 16:01:21.231198 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:21.230511    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh" event={"ID":"74161930-5277-4a96-a3ab-c263334800b2","Type":"ContainerStarted","Data":"6a56d47aaf8c7cf58abbdb43c2c60e7067a8192a86f7233885067bf3ff3742c6"}
Sep 10 16:01:21.280764 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:21.280701    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh" podStartSLOduration=1.28068082 podStartE2EDuration="1.28068082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:21.279071795 +0000 UTC m=+1160.083391056" watchObservedRunningTime="2026-09-10 16:01:21.28068082 +0000 UTC m=+1160.085000084"
Sep 10 16:01:21.562244 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:21.562119    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:21.567459 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:21.567435    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:21.790051 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:21.790018    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:01:21.790372 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:21.790348    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 10 16:01:21.790428 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:21.790397    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw" podUID="f413d5cd-e143-4982-8e07-ede979564650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 10 16:01:22.236549 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:22.236513    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:22.237530 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:22.237512    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:22.790088 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:22.790053    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 10 16:01:22.790297 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:22.790117    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw" podUID="f413d5cd-e143-4982-8e07-ede979564650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 10 16:01:23.790468 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:23.790436    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 10 16:01:23.790866 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:23.790491    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw" podUID="f413d5cd-e143-4982-8e07-ede979564650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 10 16:01:24.790303 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:24.790260    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 10 16:01:24.790484 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:24.790320    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw" podUID="f413d5cd-e143-4982-8e07-ede979564650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 10 16:01:25.790326 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:25.790291    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Sep 10 16:01:25.790501 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:25.790348    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw" podUID="f413d5cd-e143-4982-8e07-ede979564650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Sep 10 16:01:26.794218 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:26.794150    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:01:26.794638 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:26.794432    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:01:26.795090 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:26.795072    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:01:36.170004 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.169967    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"]
Sep 10 16:01:36.173544 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.173522    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:36.177257 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.177232    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b52270\"/\"auth-test-model-b52270-kserve-self-signed-certs\""
Sep 10 16:01:36.177383 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.177273    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b52270\"/\"openshift-service-ca.crt\""
Sep 10 16:01:36.177383 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.177349    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b52270\"/\"kube-root-ca.crt\""
Sep 10 16:01:36.177506 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.177254    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b52270\"/\"default-dockercfg-x4gg9\""
Sep 10 16:01:36.182143 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.182111    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"]
Sep 10 16:01:36.287363 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.287320    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-model-cache\") pod \"auth-test-model-b52270-kserve-78bcdcb595-58qjk\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:36.287555 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.287378    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-dshm\") pod \"auth-test-model-b52270-kserve-78bcdcb595-58qjk\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:36.287555 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.287480    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcs5v\" (UniqueName: \"kubernetes.io/projected/922ec163-5d92-4d61-9fdd-55175432b52d-kube-api-access-xcs5v\") pod \"auth-test-model-b52270-kserve-78bcdcb595-58qjk\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:36.287555 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.287523    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-tmp-dir\") pod \"auth-test-model-b52270-kserve-78bcdcb595-58qjk\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:36.287740 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.287558    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/922ec163-5d92-4d61-9fdd-55175432b52d-tls-certs\") pod \"auth-test-model-b52270-kserve-78bcdcb595-58qjk\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:36.287740 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.287620    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-home\") pod \"auth-test-model-b52270-kserve-78bcdcb595-58qjk\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:36.388556 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.388523    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-home\") pod \"auth-test-model-b52270-kserve-78bcdcb595-58qjk\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:36.388776 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.388567    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-model-cache\") pod \"auth-test-model-b52270-kserve-78bcdcb595-58qjk\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:36.388776 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.388634    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-dshm\") pod \"auth-test-model-b52270-kserve-78bcdcb595-58qjk\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:36.388776 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.388689    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcs5v\" (UniqueName: \"kubernetes.io/projected/922ec163-5d92-4d61-9fdd-55175432b52d-kube-api-access-xcs5v\") pod \"auth-test-model-b52270-kserve-78bcdcb595-58qjk\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:36.388776 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.388724    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-tmp-dir\") pod \"auth-test-model-b52270-kserve-78bcdcb595-58qjk\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:36.389009 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.388799    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/922ec163-5d92-4d61-9fdd-55175432b52d-tls-certs\") pod \"auth-test-model-b52270-kserve-78bcdcb595-58qjk\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:36.389066 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.389035    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-home\") pod \"auth-test-model-b52270-kserve-78bcdcb595-58qjk\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:36.389134 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.389088    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-model-cache\") pod \"auth-test-model-b52270-kserve-78bcdcb595-58qjk\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:36.389134 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.389117    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-tmp-dir\") pod \"auth-test-model-b52270-kserve-78bcdcb595-58qjk\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:36.391191 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.391126    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-dshm\") pod \"auth-test-model-b52270-kserve-78bcdcb595-58qjk\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:36.391455 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.391436    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/922ec163-5d92-4d61-9fdd-55175432b52d-tls-certs\") pod \"auth-test-model-b52270-kserve-78bcdcb595-58qjk\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:36.402082 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.402053    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcs5v\" (UniqueName: \"kubernetes.io/projected/922ec163-5d92-4d61-9fdd-55175432b52d-kube-api-access-xcs5v\") pod \"auth-test-model-b52270-kserve-78bcdcb595-58qjk\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:36.488011 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.487928    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:36.765213 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:36.765182    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"]
Sep 10 16:01:36.765355 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:01:36.765333    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod922ec163_5d92_4d61_9fdd_55175432b52d.slice/crio-b32fd23c600dd6c0c5c1759f1e7d41a30b3cd58a6d3616f7d2cb3a75d6421ddf WatchSource:0}: Error finding container b32fd23c600dd6c0c5c1759f1e7d41a30b3cd58a6d3616f7d2cb3a75d6421ddf: Status 404 returned error can't find the container with id b32fd23c600dd6c0c5c1759f1e7d41a30b3cd58a6d3616f7d2cb3a75d6421ddf
Sep 10 16:01:37.357630 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:37.357582    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk" event={"ID":"922ec163-5d92-4d61-9fdd-55175432b52d","Type":"ContainerStarted","Data":"6cf50c981b612dbff75659de24a850d7e97d159f053cd2e1198bfafc1d4a7f96"}
Sep 10 16:01:37.358098 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:37.357643    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk" event={"ID":"922ec163-5d92-4d61-9fdd-55175432b52d","Type":"ContainerStarted","Data":"b32fd23c600dd6c0c5c1759f1e7d41a30b3cd58a6d3616f7d2cb3a75d6421ddf"}
Sep 10 16:01:37.377130 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:37.377067    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk" podStartSLOduration=1.377050998 podStartE2EDuration="1.377050998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:37.37482601 +0000 UTC m=+1176.179145284" watchObservedRunningTime="2026-09-10 16:01:37.377050998 +0000 UTC m=+1176.181370259"
Sep 10 16:01:38.278381 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:38.278337    2564 patch_prober.go:28] interesting pod/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 16:01:38.278542 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:38.278403    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh" podUID="74161930-5277-4a96-a3ab-c263334800b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 16:01:40.379050 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:40.378952    2564 generic.go:356] "Generic (PLEG): container finished" podID="74161930-5277-4a96-a3ab-c263334800b2" containerID="c9f112201c8224938d8b425c4fdd486a01bd8950ee51457663f7e2f733930c02" exitCode=0
Sep 10 16:01:40.379050 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:40.379032    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh" event={"ID":"74161930-5277-4a96-a3ab-c263334800b2","Type":"ContainerDied","Data":"c9f112201c8224938d8b425c4fdd486a01bd8950ee51457663f7e2f733930c02"}
Sep 10 16:01:40.379674 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:40.379600    2564 scope.go:117] "RemoveContainer" containerID="c9f112201c8224938d8b425c4fdd486a01bd8950ee51457663f7e2f733930c02"
Sep 10 16:01:40.380990 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:40.380954    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:01:40.381086 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:40.381041    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:01:40.381132 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:40.381084    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:01:41.387380 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:41.387347    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh" event={"ID":"74161930-5277-4a96-a3ab-c263334800b2","Type":"ContainerStarted","Data":"05a22461192144830afe282b035ac4941e268640b428c167e1f862f7dd0b1678"}
Sep 10 16:01:41.562131 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:41.562095    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:41.562417 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:41.562387    2564 patch_prober.go:28] interesting pod/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 10 16:01:41.562565 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:41.562428    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh" podUID="74161930-5277-4a96-a3ab-c263334800b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 10 16:01:42.563050 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:42.563017    2564 patch_prober.go:28] interesting pod/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 10 16:01:42.563462 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:42.563067    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh" podUID="74161930-5277-4a96-a3ab-c263334800b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 10 16:01:43.562622 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:43.562582    2564 patch_prober.go:28] interesting pod/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 10 16:01:43.562826 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:43.562686    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh" podUID="74161930-5277-4a96-a3ab-c263334800b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 10 16:01:44.562712 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:44.562677    2564 patch_prober.go:28] interesting pod/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 10 16:01:44.563124 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:44.562745    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh" podUID="74161930-5277-4a96-a3ab-c263334800b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 10 16:01:45.562303 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:45.562263    2564 patch_prober.go:28] interesting pod/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 10 16:01:45.562477 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:45.562318    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh" podUID="74161930-5277-4a96-a3ab-c263334800b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 10 16:01:46.488787 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:46.488751    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:46.489246 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:46.488815    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:46.510008 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:46.509969    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:46.562582 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:46.562530    2564 patch_prober.go:28] interesting pod/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 10 16:01:46.562756 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:46.562622    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh" podUID="74161930-5277-4a96-a3ab-c263334800b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 10 16:01:47.446407 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:47.446379    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:01:47.566667 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:47.566631    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:47.567098 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:47.567021    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:01:47.567765 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:01:47.567746    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:02:02.408820 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:02.408786    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-snm4z_f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f/console-operator/1.log"
Sep 10 16:02:02.409638 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:02.409615    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-snm4z_f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f/console-operator/1.log"
Sep 10 16:02:02.479101 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:02.479075    2564 scope.go:117] "RemoveContainer" containerID="9dccc38a2a5207580cc551ea9260395e364f3423e05866618fe2b4cf5165ba8f"
Sep 10 16:02:02.488004 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:02.487986    2564 scope.go:117] "RemoveContainer" containerID="ccb57d457d1d1df631b7fd477fca05a44c89204b20d3c95a11d211f255f0baef"
Sep 10 16:02:12.464953 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.464900    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"]
Sep 10 16:02:12.469087 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.469067    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:12.472672 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.472647    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-108dbb\"/\"kube-root-ca.crt\""
Sep 10 16:02:12.472779 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.472703    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-108dbb\"/\"default-dockercfg-mzb87\""
Sep 10 16:02:12.472779 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.472731    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-108dbb\"/\"openshift-service-ca.crt\""
Sep 10 16:02:12.472779 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.472731    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-108dbb\"/\"auth-test-model-108dbb-kserve-self-signed-certs\""
Sep 10 16:02:12.483666 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.481780    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"]
Sep 10 16:02:12.593353 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.593323    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-model-cache\") pod \"auth-test-model-108dbb-kserve-cb9d96595-2phhv\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:12.593353 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.593358    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-dshm\") pod \"auth-test-model-108dbb-kserve-cb9d96595-2phhv\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:12.593581 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.593389    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c66a64a-ffe8-4358-a1be-f77f525c4631-tls-certs\") pod \"auth-test-model-108dbb-kserve-cb9d96595-2phhv\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:12.593581 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.593427    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-tmp-dir\") pod \"auth-test-model-108dbb-kserve-cb9d96595-2phhv\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:12.593581 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.593483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-home\") pod \"auth-test-model-108dbb-kserve-cb9d96595-2phhv\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:12.593581 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.593544    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v8gr\" (UniqueName: \"kubernetes.io/projected/7c66a64a-ffe8-4358-a1be-f77f525c4631-kube-api-access-7v8gr\") pod \"auth-test-model-108dbb-kserve-cb9d96595-2phhv\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:12.694515 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.694475    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-home\") pod \"auth-test-model-108dbb-kserve-cb9d96595-2phhv\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:12.694759 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.694556    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v8gr\" (UniqueName: \"kubernetes.io/projected/7c66a64a-ffe8-4358-a1be-f77f525c4631-kube-api-access-7v8gr\") pod \"auth-test-model-108dbb-kserve-cb9d96595-2phhv\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:12.694759 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.694595    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-model-cache\") pod \"auth-test-model-108dbb-kserve-cb9d96595-2phhv\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:12.694759 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.694613    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-dshm\") pod \"auth-test-model-108dbb-kserve-cb9d96595-2phhv\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:12.694759 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.694645    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c66a64a-ffe8-4358-a1be-f77f525c4631-tls-certs\") pod \"auth-test-model-108dbb-kserve-cb9d96595-2phhv\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:12.694759 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.694676    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-tmp-dir\") pod \"auth-test-model-108dbb-kserve-cb9d96595-2phhv\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:12.695037 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.694945    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-home\") pod \"auth-test-model-108dbb-kserve-cb9d96595-2phhv\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:12.695037 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.694994    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-model-cache\") pod \"auth-test-model-108dbb-kserve-cb9d96595-2phhv\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:12.695126 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.695075    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-tmp-dir\") pod \"auth-test-model-108dbb-kserve-cb9d96595-2phhv\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:12.697549 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.697523    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-dshm\") pod \"auth-test-model-108dbb-kserve-cb9d96595-2phhv\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:12.697795 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.697762    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c66a64a-ffe8-4358-a1be-f77f525c4631-tls-certs\") pod \"auth-test-model-108dbb-kserve-cb9d96595-2phhv\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:12.704639 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.704605    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v8gr\" (UniqueName: \"kubernetes.io/projected/7c66a64a-ffe8-4358-a1be-f77f525c4631-kube-api-access-7v8gr\") pod \"auth-test-model-108dbb-kserve-cb9d96595-2phhv\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:12.780011 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:12.779909    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:13.087381 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:13.087354    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"]
Sep 10 16:02:13.089352 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:02:13.089305    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c66a64a_ffe8_4358_a1be_f77f525c4631.slice/crio-5fcb532482d8b3e9091086140992a39dcc148e16e75bb576d72aa1ea3919b1ff WatchSource:0}: Error finding container 5fcb532482d8b3e9091086140992a39dcc148e16e75bb576d72aa1ea3919b1ff: Status 404 returned error can't find the container with id 5fcb532482d8b3e9091086140992a39dcc148e16e75bb576d72aa1ea3919b1ff
Sep 10 16:02:13.594994 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:13.594947    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv" event={"ID":"7c66a64a-ffe8-4358-a1be-f77f525c4631","Type":"ContainerStarted","Data":"1a2bd14fd4466aa7b1e5632e2aff2adfb8b9703825cb939e37c27efd74440975"}
Sep 10 16:02:13.595451 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:13.595005    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv" event={"ID":"7c66a64a-ffe8-4358-a1be-f77f525c4631","Type":"ContainerStarted","Data":"5fcb532482d8b3e9091086140992a39dcc148e16e75bb576d72aa1ea3919b1ff"}
Sep 10 16:02:13.617542 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:13.617490    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv" podStartSLOduration=1.617473913 podStartE2EDuration="1.617473913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:13.616210803 +0000 UTC m=+1212.420530077" watchObservedRunningTime="2026-09-10 16:02:13.617473913 +0000 UTC m=+1212.421793216"
Sep 10 16:02:22.780697 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:22.780668    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:22.781149 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:22.780714    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:22.793548 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:22.793525    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:23.672938 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:23.672908    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:02:58.627765 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:58.627682    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69972456-skzrb"]
Sep 10 16:02:58.631851 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:58.631834    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69972456-skzrb"
Sep 10 16:02:58.634405 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:58.634382    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2wghg\""
Sep 10 16:02:58.639103 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:58.639077    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69972456-skzrb"]
Sep 10 16:02:58.696925 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:58.696886    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c79a203e-5ab2-4078-bcea-184c92e55091-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-69972456-skzrb\" (UID: \"c79a203e-5ab2-4078-bcea-184c92e55091\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69972456-skzrb"
Sep 10 16:02:58.798015 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:58.797981    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c79a203e-5ab2-4078-bcea-184c92e55091-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-69972456-skzrb\" (UID: \"c79a203e-5ab2-4078-bcea-184c92e55091\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69972456-skzrb"
Sep 10 16:02:58.798823 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:58.798797    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c79a203e-5ab2-4078-bcea-184c92e55091-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-69972456-skzrb\" (UID: \"c79a203e-5ab2-4078-bcea-184c92e55091\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69972456-skzrb"
Sep 10 16:02:58.945099 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:58.945062    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69972456-skzrb"
Sep 10 16:02:59.241085 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:59.241059    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69972456-skzrb"]
Sep 10 16:02:59.243390 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:02:59.243357    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79a203e_5ab2_4078_bcea_184c92e55091.slice/crio-443847f58b213e3407799462a6cfb68b6f9759b0df47647e094ae5ab7be69c4f WatchSource:0}: Error finding container 443847f58b213e3407799462a6cfb68b6f9759b0df47647e094ae5ab7be69c4f: Status 404 returned error can't find the container with id 443847f58b213e3407799462a6cfb68b6f9759b0df47647e094ae5ab7be69c4f
Sep 10 16:02:59.893982 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:59.893882    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69972456-skzrb" event={"ID":"c79a203e-5ab2-4078-bcea-184c92e55091","Type":"ContainerStarted","Data":"c559940d367b28e9cadbd42c0bad1da97df5ddf7d3c13f5e6bbcc7fcc4b9ee2c"}
Sep 10 16:02:59.893982 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:59.893943    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69972456-skzrb" event={"ID":"c79a203e-5ab2-4078-bcea-184c92e55091","Type":"ContainerStarted","Data":"443847f58b213e3407799462a6cfb68b6f9759b0df47647e094ae5ab7be69c4f"}
Sep 10 16:02:59.914057 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:02:59.913988    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69972456-skzrb" podStartSLOduration=1.913966032 podStartE2EDuration="1.913966032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:59.91168577 +0000 UTC m=+1258.716005032" watchObservedRunningTime="2026-09-10 16:02:59.913966032 +0000 UTC m=+1258.718285294"
Sep 10 16:03:00.902181 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:03:00.902129    2564 generic.go:356] "Generic (PLEG): container finished" podID="c79a203e-5ab2-4078-bcea-184c92e55091" containerID="c559940d367b28e9cadbd42c0bad1da97df5ddf7d3c13f5e6bbcc7fcc4b9ee2c" exitCode=0
Sep 10 16:03:00.902665 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:03:00.902206    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69972456-skzrb" event={"ID":"c79a203e-5ab2-4078-bcea-184c92e55091","Type":"ContainerDied","Data":"c559940d367b28e9cadbd42c0bad1da97df5ddf7d3c13f5e6bbcc7fcc4b9ee2c"}
Sep 10 16:03:02.056335 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:03:02.056312    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69972456-skzrb"
Sep 10 16:03:02.135316 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:03:02.135282    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c79a203e-5ab2-4078-bcea-184c92e55091-maas-api-service-ca\") pod \"c79a203e-5ab2-4078-bcea-184c92e55091\" (UID: \"c79a203e-5ab2-4078-bcea-184c92e55091\") "
Sep 10 16:03:02.135672 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:03:02.135649    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c79a203e-5ab2-4078-bcea-184c92e55091-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c79a203e-5ab2-4078-bcea-184c92e55091" (UID: "c79a203e-5ab2-4078-bcea-184c92e55091"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:03:02.236889 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:03:02.236845    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c79a203e-5ab2-4078-bcea-184c92e55091-maas-api-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:03:02.517446 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:03:02.517363    2564 scope.go:117] "RemoveContainer" containerID="d87457ea7e7635599ad385975c9923c8dcf775a4297c0287a6f734cfb20da861"
Sep 10 16:03:02.915968 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:03:02.915894    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69972456-skzrb"
Sep 10 16:03:02.915968 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:03:02.915902    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69972456-skzrb" event={"ID":"c79a203e-5ab2-4078-bcea-184c92e55091","Type":"ContainerDied","Data":"443847f58b213e3407799462a6cfb68b6f9759b0df47647e094ae5ab7be69c4f"}
Sep 10 16:03:02.915968 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:03:02.915946    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="443847f58b213e3407799462a6cfb68b6f9759b0df47647e094ae5ab7be69c4f"
Sep 10 16:03:14.577177 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:03:14.577125    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69972456-skzrb"]
Sep 10 16:03:14.579731 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:03:14.579707    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-69972456-skzrb"]
Sep 10 16:03:15.673625 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:03:15.673591    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c79a203e-5ab2-4078-bcea-184c92e55091" path="/var/lib/kubelet/pods/c79a203e-5ab2-4078-bcea-184c92e55091/volumes"
Sep 10 16:03:55.886528 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:03:55.886493    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"]
Sep 10 16:03:55.887107 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:03:55.886726    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh" podUID="74161930-5277-4a96-a3ab-c263334800b2" containerName="istio-proxy" containerID="cri-o://05a22461192144830afe282b035ac4941e268640b428c167e1f862f7dd0b1678" gracePeriod=30
Sep 10 16:04:01.156925 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.156855    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:04:01.268412 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.268383    2564 generic.go:356] "Generic (PLEG): container finished" podID="74161930-5277-4a96-a3ab-c263334800b2" containerID="05a22461192144830afe282b035ac4941e268640b428c167e1f862f7dd0b1678" exitCode=0
Sep 10 16:04:01.268600 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.268436    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"
Sep 10 16:04:01.268600 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.268476    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh" event={"ID":"74161930-5277-4a96-a3ab-c263334800b2","Type":"ContainerDied","Data":"05a22461192144830afe282b035ac4941e268640b428c167e1f862f7dd0b1678"}
Sep 10 16:04:01.268600 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.268529    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh" event={"ID":"74161930-5277-4a96-a3ab-c263334800b2","Type":"ContainerDied","Data":"6a56d47aaf8c7cf58abbdb43c2c60e7067a8192a86f7233885067bf3ff3742c6"}
Sep 10 16:04:01.268600 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.268550    2564 scope.go:117] "RemoveContainer" containerID="05a22461192144830afe282b035ac4941e268640b428c167e1f862f7dd0b1678"
Sep 10 16:04:01.270905 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.270879    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7shr\" (UniqueName: \"kubernetes.io/projected/74161930-5277-4a96-a3ab-c263334800b2-kube-api-access-p7shr\") pod \"74161930-5277-4a96-a3ab-c263334800b2\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") "
Sep 10 16:04:01.271025 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.270972    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-istio-data\") pod \"74161930-5277-4a96-a3ab-c263334800b2\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") "
Sep 10 16:04:01.271025 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.271008    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-credential-socket\") pod \"74161930-5277-4a96-a3ab-c263334800b2\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") "
Sep 10 16:04:01.271148 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.271111    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74161930-5277-4a96-a3ab-c263334800b2-istio-token\") pod \"74161930-5277-4a96-a3ab-c263334800b2\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") "
Sep 10 16:04:01.271148 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.271141    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-workload-certs\") pod \"74161930-5277-4a96-a3ab-c263334800b2\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") "
Sep 10 16:04:01.271272 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.271209    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-istio-envoy\") pod \"74161930-5277-4a96-a3ab-c263334800b2\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") "
Sep 10 16:04:01.271272 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.271241    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-istio-data" (OuterVolumeSpecName: "istio-data") pod "74161930-5277-4a96-a3ab-c263334800b2" (UID: "74161930-5277-4a96-a3ab-c263334800b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:01.271272 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.271261    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74161930-5277-4a96-a3ab-c263334800b2-istio-podinfo\") pod \"74161930-5277-4a96-a3ab-c263334800b2\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") "
Sep 10 16:04:01.271417 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.271302    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74161930-5277-4a96-a3ab-c263334800b2-istiod-ca-cert\") pod \"74161930-5277-4a96-a3ab-c263334800b2\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") "
Sep 10 16:04:01.271417 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.271343    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-workload-socket\") pod \"74161930-5277-4a96-a3ab-c263334800b2\" (UID: \"74161930-5277-4a96-a3ab-c263334800b2\") "
Sep 10 16:04:01.271417 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.271344    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "74161930-5277-4a96-a3ab-c263334800b2" (UID: "74161930-5277-4a96-a3ab-c263334800b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:01.271607 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.271441    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "74161930-5277-4a96-a3ab-c263334800b2" (UID: "74161930-5277-4a96-a3ab-c263334800b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:01.272043 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.271974    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "74161930-5277-4a96-a3ab-c263334800b2" (UID: "74161930-5277-4a96-a3ab-c263334800b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:01.272210 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.272191    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-workload-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:04:01.272330 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.272317    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-istio-data\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:04:01.272456 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.272427    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-credential-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:04:01.272577 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.272326    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74161930-5277-4a96-a3ab-c263334800b2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "74161930-5277-4a96-a3ab-c263334800b2" (UID: "74161930-5277-4a96-a3ab-c263334800b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:04:01.273861 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.273837    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74161930-5277-4a96-a3ab-c263334800b2-istio-token" (OuterVolumeSpecName: "istio-token") pod "74161930-5277-4a96-a3ab-c263334800b2" (UID: "74161930-5277-4a96-a3ab-c263334800b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:04:01.274049 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.274031    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74161930-5277-4a96-a3ab-c263334800b2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "74161930-5277-4a96-a3ab-c263334800b2" (UID: "74161930-5277-4a96-a3ab-c263334800b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 16:04:01.274191 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.274135    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74161930-5277-4a96-a3ab-c263334800b2-kube-api-access-p7shr" (OuterVolumeSpecName: "kube-api-access-p7shr") pod "74161930-5277-4a96-a3ab-c263334800b2" (UID: "74161930-5277-4a96-a3ab-c263334800b2"). InnerVolumeSpecName "kube-api-access-p7shr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:04:01.274691 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.274665    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "74161930-5277-4a96-a3ab-c263334800b2" (UID: "74161930-5277-4a96-a3ab-c263334800b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:01.280945 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.280923    2564 scope.go:117] "RemoveContainer" containerID="c9f112201c8224938d8b425c4fdd486a01bd8950ee51457663f7e2f733930c02"
Sep 10 16:04:01.291192 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.291152    2564 scope.go:117] "RemoveContainer" containerID="05a22461192144830afe282b035ac4941e268640b428c167e1f862f7dd0b1678"
Sep 10 16:04:01.291485 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:04:01.291463    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a22461192144830afe282b035ac4941e268640b428c167e1f862f7dd0b1678\": container with ID starting with 05a22461192144830afe282b035ac4941e268640b428c167e1f862f7dd0b1678 not found: ID does not exist" containerID="05a22461192144830afe282b035ac4941e268640b428c167e1f862f7dd0b1678"
Sep 10 16:04:01.291532 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.291497    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a22461192144830afe282b035ac4941e268640b428c167e1f862f7dd0b1678"} err="failed to get container status \"05a22461192144830afe282b035ac4941e268640b428c167e1f862f7dd0b1678\": rpc error: code = NotFound desc = could not find container \"05a22461192144830afe282b035ac4941e268640b428c167e1f862f7dd0b1678\": container with ID starting with 05a22461192144830afe282b035ac4941e268640b428c167e1f862f7dd0b1678 not found: ID does not exist"
Sep 10 16:04:01.291532 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.291515    2564 scope.go:117] "RemoveContainer" containerID="c9f112201c8224938d8b425c4fdd486a01bd8950ee51457663f7e2f733930c02"
Sep 10 16:04:01.291789 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:04:01.291772    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f112201c8224938d8b425c4fdd486a01bd8950ee51457663f7e2f733930c02\": container with ID starting with c9f112201c8224938d8b425c4fdd486a01bd8950ee51457663f7e2f733930c02 not found: ID does not exist" containerID="c9f112201c8224938d8b425c4fdd486a01bd8950ee51457663f7e2f733930c02"
Sep 10 16:04:01.291853 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.291794    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f112201c8224938d8b425c4fdd486a01bd8950ee51457663f7e2f733930c02"} err="failed to get container status \"c9f112201c8224938d8b425c4fdd486a01bd8950ee51457663f7e2f733930c02\": rpc error: code = NotFound desc = could not find container \"c9f112201c8224938d8b425c4fdd486a01bd8950ee51457663f7e2f733930c02\": container with ID starting with c9f112201c8224938d8b425c4fdd486a01bd8950ee51457663f7e2f733930c02 not found: ID does not exist"
Sep 10 16:04:01.373779 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.373740    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74161930-5277-4a96-a3ab-c263334800b2-istio-podinfo\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:04:01.373779 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.373771    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74161930-5277-4a96-a3ab-c263334800b2-istiod-ca-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:04:01.373779 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.373782    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-workload-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:04:01.374028 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.373792    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7shr\" (UniqueName: \"kubernetes.io/projected/74161930-5277-4a96-a3ab-c263334800b2-kube-api-access-p7shr\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:04:01.374028 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.373801    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74161930-5277-4a96-a3ab-c263334800b2-istio-token\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:04:01.374028 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.373809    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74161930-5277-4a96-a3ab-c263334800b2-istio-envoy\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:04:01.596332 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.596301    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"]
Sep 10 16:04:01.604149 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.604123    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-69972456-openshift-default-76fcb6cd44-dnrbh"]
Sep 10 16:04:01.678896 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:01.678857    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74161930-5277-4a96-a3ab-c263334800b2" path="/var/lib/kubelet/pods/74161930-5277-4a96-a3ab-c263334800b2/volumes"
Sep 10 16:04:02.539447 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:02.539417    2564 scope.go:117] "RemoveContainer" containerID="f0e2212dcc0530c25b8e3ed5dd293d3e932c08b39f557f320396628eedb7dd92"
Sep 10 16:04:02.548722 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:02.548702    2564 scope.go:117] "RemoveContainer" containerID="deb47ec555ac6212fc1ca9563257fd574fe6720d2131bae065793b1d4d379039"
Sep 10 16:04:40.729247 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:40.729217    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"]
Sep 10 16:04:41.665730 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:41.665696    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"]
Sep 10 16:04:53.721881 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.721847    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"]
Sep 10 16:04:53.722363 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.722330    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74161930-5277-4a96-a3ab-c263334800b2" containerName="istio-proxy"
Sep 10 16:04:53.722363 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.722345    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="74161930-5277-4a96-a3ab-c263334800b2" containerName="istio-proxy"
Sep 10 16:04:53.722363 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.722354    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74161930-5277-4a96-a3ab-c263334800b2" containerName="istio-proxy"
Sep 10 16:04:53.722363 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.722360    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="74161930-5277-4a96-a3ab-c263334800b2" containerName="istio-proxy"
Sep 10 16:04:53.722498 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.722367    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c79a203e-5ab2-4078-bcea-184c92e55091" containerName="revoke-keys"
Sep 10 16:04:53.722498 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.722373    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c79a203e-5ab2-4078-bcea-184c92e55091" containerName="revoke-keys"
Sep 10 16:04:53.722498 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.722454    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c79a203e-5ab2-4078-bcea-184c92e55091" containerName="revoke-keys"
Sep 10 16:04:53.722498 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.722463    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="74161930-5277-4a96-a3ab-c263334800b2" containerName="istio-proxy"
Sep 10 16:04:53.722498 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.722470    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="74161930-5277-4a96-a3ab-c263334800b2" containerName="istio-proxy"
Sep 10 16:04:53.726009 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.725988    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.728763 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.728732    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ea0f68-openshift-default-dockercfg-dzwzm\""
Sep 10 16:04:53.740759 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.740732    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"]
Sep 10 16:04:53.836270 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.836223    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0531a19-b416-414c-aea2-430cc11dbb23-istio-podinfo\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.836477 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.836307    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-workload-socket\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.836477 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.836442    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-credential-socket\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.836594 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.836505    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-workload-certs\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.836594 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.836542    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0531a19-b416-414c-aea2-430cc11dbb23-istio-token\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.836679 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.836590    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-istio-envoy\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.836679 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.836664    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sfhj\" (UniqueName: \"kubernetes.io/projected/c0531a19-b416-414c-aea2-430cc11dbb23-kube-api-access-6sfhj\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.836756 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.836691    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-istio-data\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.836756 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.836728    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0531a19-b416-414c-aea2-430cc11dbb23-istiod-ca-cert\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.937882 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.937843    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-credential-socket\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.938066 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.937891    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-workload-certs\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.938066 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.937914    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0531a19-b416-414c-aea2-430cc11dbb23-istio-token\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.938066 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.938037    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-istio-envoy\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.938288 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.938114    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfhj\" (UniqueName: \"kubernetes.io/projected/c0531a19-b416-414c-aea2-430cc11dbb23-kube-api-access-6sfhj\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.938288 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.938139    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-istio-data\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.938288 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.938208    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0531a19-b416-414c-aea2-430cc11dbb23-istiod-ca-cert\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.938438 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.938301    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0531a19-b416-414c-aea2-430cc11dbb23-istio-podinfo\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.938438 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.938339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-workload-socket\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.938438 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.938368    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-credential-socket\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.938564 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.938506    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-istio-data\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.938631 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.938610    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-workload-certs\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.938692 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.938652    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-workload-socket\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.938864 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.938838    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0531a19-b416-414c-aea2-430cc11dbb23-istiod-ca-cert\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.940566 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.940539    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-istio-envoy\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.940787 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.940772    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0531a19-b416-414c-aea2-430cc11dbb23-istio-podinfo\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.947993 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.947974    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0531a19-b416-414c-aea2-430cc11dbb23-istio-token\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:53.948288 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:53.948271    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sfhj\" (UniqueName: \"kubernetes.io/projected/c0531a19-b416-414c-aea2-430cc11dbb23-kube-api-access-6sfhj\") pod \"e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") " pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:54.037801 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:54.037697    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:54.317089 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:54.316982    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"]
Sep 10 16:04:54.320116 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:04:54.320092    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0531a19_b416_414c_aea2_430cc11dbb23.slice/crio-14f130177028d70f1ef878dc551c92bb2d9eccdfe8639af7bb619d82c32dc032 WatchSource:0}: Error finding container 14f130177028d70f1ef878dc551c92bb2d9eccdfe8639af7bb619d82c32dc032: Status 404 returned error can't find the container with id 14f130177028d70f1ef878dc551c92bb2d9eccdfe8639af7bb619d82c32dc032
Sep 10 16:04:54.322485 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:54.322451    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:04:54.322562 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:54.322520    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:04:54.322562 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:54.322550    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:04:54.590467 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:54.590302    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj" event={"ID":"c0531a19-b416-414c-aea2-430cc11dbb23","Type":"ContainerStarted","Data":"005024856a8b83fa662904f051c8c5104b14a5c3ea75164e5313f977df96050c"}
Sep 10 16:04:54.590467 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:54.590377    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj" event={"ID":"c0531a19-b416-414c-aea2-430cc11dbb23","Type":"ContainerStarted","Data":"14f130177028d70f1ef878dc551c92bb2d9eccdfe8639af7bb619d82c32dc032"}
Sep 10 16:04:54.614972 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:54.614914    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj" podStartSLOduration=1.61489612 podStartE2EDuration="1.61489612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:04:54.613045414 +0000 UTC m=+1373.417364678" watchObservedRunningTime="2026-09-10 16:04:54.61489612 +0000 UTC m=+1373.419215381"
Sep 10 16:04:55.038577 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:55.038538    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:55.043980 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:55.043952    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:55.596660 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:55.596631    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:55.597722 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:55.597693    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:04:55.749533 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:55.749489    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk" podUID="922ec163-5d92-4d61-9fdd-55175432b52d" containerName="main" containerID="cri-o://6cf50c981b612dbff75659de24a850d7e97d159f053cd2e1198bfafc1d4a7f96" gracePeriod=45
Sep 10 16:04:56.019451 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.019427    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:04:56.061922 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.061888    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-tmp-dir\") pod \"922ec163-5d92-4d61-9fdd-55175432b52d\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") "
Sep 10 16:04:56.062432 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.061931    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcs5v\" (UniqueName: \"kubernetes.io/projected/922ec163-5d92-4d61-9fdd-55175432b52d-kube-api-access-xcs5v\") pod \"922ec163-5d92-4d61-9fdd-55175432b52d\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") "
Sep 10 16:04:56.062432 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.062013    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-model-cache\") pod \"922ec163-5d92-4d61-9fdd-55175432b52d\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") "
Sep 10 16:04:56.062432 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.062146    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "922ec163-5d92-4d61-9fdd-55175432b52d" (UID: "922ec163-5d92-4d61-9fdd-55175432b52d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:56.062432 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.062201    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-dshm\") pod \"922ec163-5d92-4d61-9fdd-55175432b52d\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") "
Sep 10 16:04:56.062432 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.062275    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/922ec163-5d92-4d61-9fdd-55175432b52d-tls-certs\") pod \"922ec163-5d92-4d61-9fdd-55175432b52d\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") "
Sep 10 16:04:56.062432 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.062296    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-model-cache" (OuterVolumeSpecName: "model-cache") pod "922ec163-5d92-4d61-9fdd-55175432b52d" (UID: "922ec163-5d92-4d61-9fdd-55175432b52d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:56.062432 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.062327    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-home\") pod \"922ec163-5d92-4d61-9fdd-55175432b52d\" (UID: \"922ec163-5d92-4d61-9fdd-55175432b52d\") "
Sep 10 16:04:56.062887 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.062540    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-home" (OuterVolumeSpecName: "home") pod "922ec163-5d92-4d61-9fdd-55175432b52d" (UID: "922ec163-5d92-4d61-9fdd-55175432b52d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:56.062887 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.062745    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-model-cache\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:04:56.062887 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.062766    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-home\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:04:56.062887 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.062779    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-tmp-dir\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:04:56.064581 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.064554    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/922ec163-5d92-4d61-9fdd-55175432b52d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "922ec163-5d92-4d61-9fdd-55175432b52d" (UID: "922ec163-5d92-4d61-9fdd-55175432b52d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:04:56.064691 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.064554    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-dshm" (OuterVolumeSpecName: "dshm") pod "922ec163-5d92-4d61-9fdd-55175432b52d" (UID: "922ec163-5d92-4d61-9fdd-55175432b52d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:56.064752 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.064686    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922ec163-5d92-4d61-9fdd-55175432b52d-kube-api-access-xcs5v" (OuterVolumeSpecName: "kube-api-access-xcs5v") pod "922ec163-5d92-4d61-9fdd-55175432b52d" (UID: "922ec163-5d92-4d61-9fdd-55175432b52d"). InnerVolumeSpecName "kube-api-access-xcs5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:04:56.164092 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.164006    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/922ec163-5d92-4d61-9fdd-55175432b52d-tls-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:04:56.164092 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.164034    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcs5v\" (UniqueName: \"kubernetes.io/projected/922ec163-5d92-4d61-9fdd-55175432b52d-kube-api-access-xcs5v\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:04:56.164092 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.164044    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/922ec163-5d92-4d61-9fdd-55175432b52d-dshm\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:04:56.605958 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.605914    2564 generic.go:356] "Generic (PLEG): container finished" podID="922ec163-5d92-4d61-9fdd-55175432b52d" containerID="6cf50c981b612dbff75659de24a850d7e97d159f053cd2e1198bfafc1d4a7f96" exitCode=0
Sep 10 16:04:56.606176 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.605975    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"
Sep 10 16:04:56.606176 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.605993    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk" event={"ID":"922ec163-5d92-4d61-9fdd-55175432b52d","Type":"ContainerDied","Data":"6cf50c981b612dbff75659de24a850d7e97d159f053cd2e1198bfafc1d4a7f96"}
Sep 10 16:04:56.606176 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.606034    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk" event={"ID":"922ec163-5d92-4d61-9fdd-55175432b52d","Type":"ContainerDied","Data":"b32fd23c600dd6c0c5c1759f1e7d41a30b3cd58a6d3616f7d2cb3a75d6421ddf"}
Sep 10 16:04:56.606176 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.606051    2564 scope.go:117] "RemoveContainer" containerID="6cf50c981b612dbff75659de24a850d7e97d159f053cd2e1198bfafc1d4a7f96"
Sep 10 16:04:56.616851 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.616833    2564 scope.go:117] "RemoveContainer" containerID="6cf50c981b612dbff75659de24a850d7e97d159f053cd2e1198bfafc1d4a7f96"
Sep 10 16:04:56.617188 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:04:56.617142    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cf50c981b612dbff75659de24a850d7e97d159f053cd2e1198bfafc1d4a7f96\": container with ID starting with 6cf50c981b612dbff75659de24a850d7e97d159f053cd2e1198bfafc1d4a7f96 not found: ID does not exist" containerID="6cf50c981b612dbff75659de24a850d7e97d159f053cd2e1198bfafc1d4a7f96"
Sep 10 16:04:56.617256 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.617199    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cf50c981b612dbff75659de24a850d7e97d159f053cd2e1198bfafc1d4a7f96"} err="failed to get container status \"6cf50c981b612dbff75659de24a850d7e97d159f053cd2e1198bfafc1d4a7f96\": rpc error: code = NotFound desc = could not find container \"6cf50c981b612dbff75659de24a850d7e97d159f053cd2e1198bfafc1d4a7f96\": container with ID starting with 6cf50c981b612dbff75659de24a850d7e97d159f053cd2e1198bfafc1d4a7f96 not found: ID does not exist"
Sep 10 16:04:56.636912 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.636889    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"]
Sep 10 16:04:56.647495 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.647474    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b52270/auth-test-model-b52270-kserve-78bcdcb595-58qjk"]
Sep 10 16:04:56.696994 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.696932    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv" podUID="7c66a64a-ffe8-4358-a1be-f77f525c4631" containerName="main" containerID="cri-o://1a2bd14fd4466aa7b1e5632e2aff2adfb8b9703825cb939e37c27efd74440975" gracePeriod=45
Sep 10 16:04:56.937955 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.937933    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:04:56.973110 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.973084    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-dshm\") pod \"7c66a64a-ffe8-4358-a1be-f77f525c4631\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") "
Sep 10 16:04:56.973301 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.973121    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-home\") pod \"7c66a64a-ffe8-4358-a1be-f77f525c4631\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") "
Sep 10 16:04:56.973301 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.973237    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c66a64a-ffe8-4358-a1be-f77f525c4631-tls-certs\") pod \"7c66a64a-ffe8-4358-a1be-f77f525c4631\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") "
Sep 10 16:04:56.973428 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.973306    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v8gr\" (UniqueName: \"kubernetes.io/projected/7c66a64a-ffe8-4358-a1be-f77f525c4631-kube-api-access-7v8gr\") pod \"7c66a64a-ffe8-4358-a1be-f77f525c4631\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") "
Sep 10 16:04:56.973428 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.973375    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-model-cache\") pod \"7c66a64a-ffe8-4358-a1be-f77f525c4631\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") "
Sep 10 16:04:56.973428 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.973398    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-home" (OuterVolumeSpecName: "home") pod "7c66a64a-ffe8-4358-a1be-f77f525c4631" (UID: "7c66a64a-ffe8-4358-a1be-f77f525c4631"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:56.973428 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.973405    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-tmp-dir\") pod \"7c66a64a-ffe8-4358-a1be-f77f525c4631\" (UID: \"7c66a64a-ffe8-4358-a1be-f77f525c4631\") "
Sep 10 16:04:56.973627 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.973573    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c66a64a-ffe8-4358-a1be-f77f525c4631" (UID: "7c66a64a-ffe8-4358-a1be-f77f525c4631"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:56.973773 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.973743    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c66a64a-ffe8-4358-a1be-f77f525c4631" (UID: "7c66a64a-ffe8-4358-a1be-f77f525c4631"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:56.973937 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.973918    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-model-cache\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:04:56.974005 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.973942    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-tmp-dir\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:04:56.974005 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.973955    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-home\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:04:56.975775 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.975749    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c66a64a-ffe8-4358-a1be-f77f525c4631-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c66a64a-ffe8-4358-a1be-f77f525c4631" (UID: "7c66a64a-ffe8-4358-a1be-f77f525c4631"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:04:56.975862 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.975849    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-dshm" (OuterVolumeSpecName: "dshm") pod "7c66a64a-ffe8-4358-a1be-f77f525c4631" (UID: "7c66a64a-ffe8-4358-a1be-f77f525c4631"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:56.976072 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:56.976049    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c66a64a-ffe8-4358-a1be-f77f525c4631-kube-api-access-7v8gr" (OuterVolumeSpecName: "kube-api-access-7v8gr") pod "7c66a64a-ffe8-4358-a1be-f77f525c4631" (UID: "7c66a64a-ffe8-4358-a1be-f77f525c4631"). InnerVolumeSpecName "kube-api-access-7v8gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:04:57.074995 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:57.074965    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c66a64a-ffe8-4358-a1be-f77f525c4631-dshm\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:04:57.074995 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:57.074994    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c66a64a-ffe8-4358-a1be-f77f525c4631-tls-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:04:57.075427 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:57.075008    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v8gr\" (UniqueName: \"kubernetes.io/projected/7c66a64a-ffe8-4358-a1be-f77f525c4631-kube-api-access-7v8gr\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:04:57.613231 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:57.613194    2564 generic.go:356] "Generic (PLEG): container finished" podID="7c66a64a-ffe8-4358-a1be-f77f525c4631" containerID="1a2bd14fd4466aa7b1e5632e2aff2adfb8b9703825cb939e37c27efd74440975" exitCode=0
Sep 10 16:04:57.613444 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:57.613246    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"
Sep 10 16:04:57.613444 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:57.613280    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv" event={"ID":"7c66a64a-ffe8-4358-a1be-f77f525c4631","Type":"ContainerDied","Data":"1a2bd14fd4466aa7b1e5632e2aff2adfb8b9703825cb939e37c27efd74440975"}
Sep 10 16:04:57.613444 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:57.613320    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv" event={"ID":"7c66a64a-ffe8-4358-a1be-f77f525c4631","Type":"ContainerDied","Data":"5fcb532482d8b3e9091086140992a39dcc148e16e75bb576d72aa1ea3919b1ff"}
Sep 10 16:04:57.613444 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:57.613337    2564 scope.go:117] "RemoveContainer" containerID="1a2bd14fd4466aa7b1e5632e2aff2adfb8b9703825cb939e37c27efd74440975"
Sep 10 16:04:57.623314 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:57.623300    2564 scope.go:117] "RemoveContainer" containerID="1a2bd14fd4466aa7b1e5632e2aff2adfb8b9703825cb939e37c27efd74440975"
Sep 10 16:04:57.623565 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:04:57.623550    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a2bd14fd4466aa7b1e5632e2aff2adfb8b9703825cb939e37c27efd74440975\": container with ID starting with 1a2bd14fd4466aa7b1e5632e2aff2adfb8b9703825cb939e37c27efd74440975 not found: ID does not exist" containerID="1a2bd14fd4466aa7b1e5632e2aff2adfb8b9703825cb939e37c27efd74440975"
Sep 10 16:04:57.623617 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:57.623572    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a2bd14fd4466aa7b1e5632e2aff2adfb8b9703825cb939e37c27efd74440975"} err="failed to get container status \"1a2bd14fd4466aa7b1e5632e2aff2adfb8b9703825cb939e37c27efd74440975\": rpc error: code = NotFound desc = could not find container \"1a2bd14fd4466aa7b1e5632e2aff2adfb8b9703825cb939e37c27efd74440975\": container with ID starting with 1a2bd14fd4466aa7b1e5632e2aff2adfb8b9703825cb939e37c27efd74440975 not found: ID does not exist"
Sep 10 16:04:57.647468 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:57.647440    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"]
Sep 10 16:04:57.652824 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:57.652800    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-108dbb/auth-test-model-108dbb-kserve-cb9d96595-2phhv"]
Sep 10 16:04:57.674248 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:57.674217    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c66a64a-ffe8-4358-a1be-f77f525c4631" path="/var/lib/kubelet/pods/7c66a64a-ffe8-4358-a1be-f77f525c4631/volumes"
Sep 10 16:04:57.674674 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:04:57.674656    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="922ec163-5d92-4d61-9fdd-55175432b52d" path="/var/lib/kubelet/pods/922ec163-5d92-4d61-9fdd-55175432b52d/volumes"
Sep 10 16:05:02.578604 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:02.578572    2564 scope.go:117] "RemoveContainer" containerID="0ae9f07be9f36b41cda04cd292d613c40819e585eb2b0a334dd9483729100d9d"
Sep 10 16:05:11.564731 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.564696    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"]
Sep 10 16:05:11.565440 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.565414    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="922ec163-5d92-4d61-9fdd-55175432b52d" containerName="main"
Sep 10 16:05:11.565440 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.565440    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="922ec163-5d92-4d61-9fdd-55175432b52d" containerName="main"
Sep 10 16:05:11.565609 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.565490    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c66a64a-ffe8-4358-a1be-f77f525c4631" containerName="main"
Sep 10 16:05:11.565609 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.565498    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c66a64a-ffe8-4358-a1be-f77f525c4631" containerName="main"
Sep 10 16:05:11.565726 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.565611    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c66a64a-ffe8-4358-a1be-f77f525c4631" containerName="main"
Sep 10 16:05:11.565726 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.565629    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="922ec163-5d92-4d61-9fdd-55175432b52d" containerName="main"
Sep 10 16:05:11.570487 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.570463    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:11.574057 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.573936    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b52270\"/\"openshift-service-ca.crt\""
Sep 10 16:05:11.575250 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.575002    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b52270\"/\"kube-root-ca.crt\""
Sep 10 16:05:11.575250 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.575039    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b52270\"/\"default-dockercfg-x4gg9\""
Sep 10 16:05:11.575250 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.575079    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b52270\"/\"sub-test-model-b52270-kserve-self-signed-certs\""
Sep 10 16:05:11.587053 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.587027    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"]
Sep 10 16:05:11.734401 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.734362    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-model-cache\") pod \"sub-test-model-b52270-kserve-666bdf5d4c-gkmjg\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") " pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:11.734606 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.734554    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp6bp\" (UniqueName: \"kubernetes.io/projected/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-kube-api-access-zp6bp\") pod \"sub-test-model-b52270-kserve-666bdf5d4c-gkmjg\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") " pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:11.734698 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.734592    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-tls-certs\") pod \"sub-test-model-b52270-kserve-666bdf5d4c-gkmjg\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") " pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:11.734698 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.734638    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-tmp-dir\") pod \"sub-test-model-b52270-kserve-666bdf5d4c-gkmjg\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") " pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:11.734799 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.734699    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-dshm\") pod \"sub-test-model-b52270-kserve-666bdf5d4c-gkmjg\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") " pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:11.734799 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.734789    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-home\") pod \"sub-test-model-b52270-kserve-666bdf5d4c-gkmjg\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") " pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:11.837461 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.837361    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp6bp\" (UniqueName: \"kubernetes.io/projected/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-kube-api-access-zp6bp\") pod \"sub-test-model-b52270-kserve-666bdf5d4c-gkmjg\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") " pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:11.837461 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.837410    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-tls-certs\") pod \"sub-test-model-b52270-kserve-666bdf5d4c-gkmjg\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") " pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:11.837461 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.837428    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-tmp-dir\") pod \"sub-test-model-b52270-kserve-666bdf5d4c-gkmjg\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") " pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:11.837461 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.837457    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-dshm\") pod \"sub-test-model-b52270-kserve-666bdf5d4c-gkmjg\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") " pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:11.837806 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.837527    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-home\") pod \"sub-test-model-b52270-kserve-666bdf5d4c-gkmjg\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") " pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:11.837806 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.837559    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-model-cache\") pod \"sub-test-model-b52270-kserve-666bdf5d4c-gkmjg\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") " pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:11.837917 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.837874    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-tmp-dir\") pod \"sub-test-model-b52270-kserve-666bdf5d4c-gkmjg\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") " pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:11.837974 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.837934    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-model-cache\") pod \"sub-test-model-b52270-kserve-666bdf5d4c-gkmjg\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") " pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:11.837974 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.837961    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-home\") pod \"sub-test-model-b52270-kserve-666bdf5d4c-gkmjg\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") " pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:11.840112 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.840080    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-dshm\") pod \"sub-test-model-b52270-kserve-666bdf5d4c-gkmjg\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") " pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:11.840386 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.840366    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-tls-certs\") pod \"sub-test-model-b52270-kserve-666bdf5d4c-gkmjg\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") " pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:11.847687 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.847660    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp6bp\" (UniqueName: \"kubernetes.io/projected/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-kube-api-access-zp6bp\") pod \"sub-test-model-b52270-kserve-666bdf5d4c-gkmjg\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") " pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:11.883343 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:11.883308    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:12.187782 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:12.187753    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"]
Sep 10 16:05:12.189789 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:05:12.189761    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9014c790_cf33_4579_9eba_ae8b2bd0ddd3.slice/crio-b31485c78f5137f1902135f74065713e72a1bd023b7dc431ecc0acdaf0fbc16f WatchSource:0}: Error finding container b31485c78f5137f1902135f74065713e72a1bd023b7dc431ecc0acdaf0fbc16f: Status 404 returned error can't find the container with id b31485c78f5137f1902135f74065713e72a1bd023b7dc431ecc0acdaf0fbc16f
Sep 10 16:05:12.712447 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:12.712408    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg" event={"ID":"9014c790-cf33-4579-9eba-ae8b2bd0ddd3","Type":"ContainerStarted","Data":"fe8226743c7303f74e47e3a91c7a3cd3de1a107f98ea366170f7262eab649029"}
Sep 10 16:05:12.712932 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:12.712459    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg" event={"ID":"9014c790-cf33-4579-9eba-ae8b2bd0ddd3","Type":"ContainerStarted","Data":"b31485c78f5137f1902135f74065713e72a1bd023b7dc431ecc0acdaf0fbc16f"}
Sep 10 16:05:14.725706 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:14.725674    2564 generic.go:356] "Generic (PLEG): container finished" podID="c0531a19-b416-414c-aea2-430cc11dbb23" containerID="005024856a8b83fa662904f051c8c5104b14a5c3ea75164e5313f977df96050c" exitCode=0
Sep 10 16:05:14.726128 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:14.725758    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj" event={"ID":"c0531a19-b416-414c-aea2-430cc11dbb23","Type":"ContainerDied","Data":"005024856a8b83fa662904f051c8c5104b14a5c3ea75164e5313f977df96050c"}
Sep 10 16:05:14.726234 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:14.726221    2564 scope.go:117] "RemoveContainer" containerID="005024856a8b83fa662904f051c8c5104b14a5c3ea75164e5313f977df96050c"
Sep 10 16:05:14.727380 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:14.727346    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:05:14.727480 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:14.727434    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:05:14.727549 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:14.727482    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:05:14.753291 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:14.753243    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg" podStartSLOduration=3.753222824 podStartE2EDuration="3.753222824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:05:12.73073034 +0000 UTC m=+1391.535049600" watchObservedRunningTime="2026-09-10 16:05:14.753222824 +0000 UTC m=+1393.557542087"
Sep 10 16:05:15.734016 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:15.733978    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj" event={"ID":"c0531a19-b416-414c-aea2-430cc11dbb23","Type":"ContainerStarted","Data":"c826fdef974e7549066f9854a0869facdd6305086186a470ca6e7a541dd7dd4c"}
Sep 10 16:05:16.038811 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:16.038713    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:05:16.039021 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:16.038996    2564 patch_prober.go:28] interesting pod/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 10 16:05:16.039087 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:16.039050    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj" podUID="c0531a19-b416-414c-aea2-430cc11dbb23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 10 16:05:17.038796 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:17.038760    2564 patch_prober.go:28] interesting pod/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 10 16:05:17.039220 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:17.038854    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj" podUID="c0531a19-b416-414c-aea2-430cc11dbb23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 10 16:05:18.038766 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:18.038729    2564 patch_prober.go:28] interesting pod/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 10 16:05:18.038959 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:18.038787    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj" podUID="c0531a19-b416-414c-aea2-430cc11dbb23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 10 16:05:19.038518 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:19.038482    2564 patch_prober.go:28] interesting pod/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 10 16:05:19.038696 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:19.038541    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj" podUID="c0531a19-b416-414c-aea2-430cc11dbb23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 10 16:05:20.038715 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:20.038671    2564 patch_prober.go:28] interesting pod/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 10 16:05:20.039222 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:20.038735    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj" podUID="c0531a19-b416-414c-aea2-430cc11dbb23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 10 16:05:21.111723 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:21.111690    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:05:21.112223 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:21.112060    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:05:21.112893 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:21.112876    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:05:21.883821 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:21.883791    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:21.883993 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:21.883838    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:21.897000 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:21.896975    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:22.327538 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.327481    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"]
Sep 10 16:05:22.331436 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.331414    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.340370 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.340350    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2ab381-openshift-default-dockercfg-gj2mk\""
Sep 10 16:05:22.350606 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.350582    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"]
Sep 10 16:05:22.359419 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.359383    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-workload-socket\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.359554 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.359450    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d171cbb-6111-4067-b134-b9016a99b03f-istio-podinfo\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.359554 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.359528    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d171cbb-6111-4067-b134-b9016a99b03f-istiod-ca-cert\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.359662 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.359561    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxqzh\" (UniqueName: \"kubernetes.io/projected/2d171cbb-6111-4067-b134-b9016a99b03f-kube-api-access-vxqzh\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.359662 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.359606    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-istio-envoy\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.359662 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.359640    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d171cbb-6111-4067-b134-b9016a99b03f-istio-token\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.359827 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.359683    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-istio-data\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.359827 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.359725    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-workload-certs\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.359827 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.359782    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-credential-socket\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.461036 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.460991    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-credential-socket\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.461272 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.461117    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-workload-socket\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.461272 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.461232    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d171cbb-6111-4067-b134-b9016a99b03f-istio-podinfo\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.461394 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.461316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d171cbb-6111-4067-b134-b9016a99b03f-istiod-ca-cert\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.461394 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.461355    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxqzh\" (UniqueName: \"kubernetes.io/projected/2d171cbb-6111-4067-b134-b9016a99b03f-kube-api-access-vxqzh\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.461510 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.461400    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-istio-envoy\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.461510 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.461439    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d171cbb-6111-4067-b134-b9016a99b03f-istio-token\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.461510 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.461483    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-istio-data\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.461679 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.461515    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-workload-certs\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.461740 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.461695    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-credential-socket\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.461892 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.461867    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-workload-certs\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.462876 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.462845    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-istio-data\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.463293 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.463273    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d171cbb-6111-4067-b134-b9016a99b03f-istiod-ca-cert\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.463451 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.463429    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-workload-socket\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.465235 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.465210    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d171cbb-6111-4067-b134-b9016a99b03f-istio-podinfo\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.465319 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.465213    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-istio-envoy\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.471438 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.471419    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d171cbb-6111-4067-b134-b9016a99b03f-istio-token\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.471517 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.471452    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxqzh\" (UniqueName: \"kubernetes.io/projected/2d171cbb-6111-4067-b134-b9016a99b03f-kube-api-access-vxqzh\") pod \"e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") " pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.643240 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.643121    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:22.791774 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.791742    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:05:22.926009 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.925970    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"]
Sep 10 16:05:22.929451 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:05:22.929424    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d171cbb_6111_4067_b134_b9016a99b03f.slice/crio-a714d98d70f074d0071e373df22f554225c50dd0eff16a590c278496cae50e98 WatchSource:0}: Error finding container a714d98d70f074d0071e373df22f554225c50dd0eff16a590c278496cae50e98: Status 404 returned error can't find the container with id a714d98d70f074d0071e373df22f554225c50dd0eff16a590c278496cae50e98
Sep 10 16:05:22.932917 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.932884    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:05:22.933022 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.932951    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:05:22.933022 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:22.932985    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:05:23.787667 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:23.787624    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr" event={"ID":"2d171cbb-6111-4067-b134-b9016a99b03f","Type":"ContainerStarted","Data":"07128b69afe2e5f8657d61d2c622c6dfe7ae6ebcd15ccf0f9f09c63ebc15f773"}
Sep 10 16:05:23.788096 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:23.787681    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr" event={"ID":"2d171cbb-6111-4067-b134-b9016a99b03f","Type":"ContainerStarted","Data":"a714d98d70f074d0071e373df22f554225c50dd0eff16a590c278496cae50e98"}
Sep 10 16:05:23.810747 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:23.810694    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr" podStartSLOduration=1.810679369 podStartE2EDuration="1.810679369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:05:23.809090838 +0000 UTC m=+1402.613410100" watchObservedRunningTime="2026-09-10 16:05:23.810679369 +0000 UTC m=+1402.614998630"
Sep 10 16:05:24.644360 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:24.644324    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:24.649848 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:24.649823    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:24.793917 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:24.793888    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:24.795074 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:24.795055    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:35.889959 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:35.889927    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"]
Sep 10 16:05:35.893655 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:35.893630    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:35.899515 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:35.899342    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-108dbb\"/\"openshift-service-ca.crt\""
Sep 10 16:05:35.899515 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:35.899367    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-108dbb\"/\"sub-test-model-108dbb-kserve-self-signed-certs\""
Sep 10 16:05:35.899515 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:35.899351    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-108dbb\"/\"default-dockercfg-mzb87\""
Sep 10 16:05:35.900099 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:35.900075    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-108dbb\"/\"kube-root-ca.crt\""
Sep 10 16:05:35.916177 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:35.910252    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"]
Sep 10 16:05:36.011635 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.011596    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/529b10fb-7718-4c71-a2e4-44358d648c2d-tls-certs\") pod \"sub-test-model-108dbb-kserve-c89d59b9-kk99j\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:36.012019 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.011996    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-model-cache\") pod \"sub-test-model-108dbb-kserve-c89d59b9-kk99j\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:36.013699 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.012271    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfrc4\" (UniqueName: \"kubernetes.io/projected/529b10fb-7718-4c71-a2e4-44358d648c2d-kube-api-access-pfrc4\") pod \"sub-test-model-108dbb-kserve-c89d59b9-kk99j\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:36.013927 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.013906    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-home\") pod \"sub-test-model-108dbb-kserve-c89d59b9-kk99j\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:36.014148 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.014122    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-tmp-dir\") pod \"sub-test-model-108dbb-kserve-c89d59b9-kk99j\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:36.014293 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.014200    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-dshm\") pod \"sub-test-model-108dbb-kserve-c89d59b9-kk99j\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:36.114915 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.114882    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-model-cache\") pod \"sub-test-model-108dbb-kserve-c89d59b9-kk99j\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:36.115123 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.114945    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfrc4\" (UniqueName: \"kubernetes.io/projected/529b10fb-7718-4c71-a2e4-44358d648c2d-kube-api-access-pfrc4\") pod \"sub-test-model-108dbb-kserve-c89d59b9-kk99j\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:36.115123 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.114974    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-home\") pod \"sub-test-model-108dbb-kserve-c89d59b9-kk99j\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:36.115123 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.115025    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-tmp-dir\") pod \"sub-test-model-108dbb-kserve-c89d59b9-kk99j\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:36.115123 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.115040    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-dshm\") pod \"sub-test-model-108dbb-kserve-c89d59b9-kk99j\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:36.115123 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.115098    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/529b10fb-7718-4c71-a2e4-44358d648c2d-tls-certs\") pod \"sub-test-model-108dbb-kserve-c89d59b9-kk99j\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:36.115460 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.115406    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-home\") pod \"sub-test-model-108dbb-kserve-c89d59b9-kk99j\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:36.115460 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.115404    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-model-cache\") pod \"sub-test-model-108dbb-kserve-c89d59b9-kk99j\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:36.115573 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.115466    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-tmp-dir\") pod \"sub-test-model-108dbb-kserve-c89d59b9-kk99j\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:36.117727 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.117700    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-dshm\") pod \"sub-test-model-108dbb-kserve-c89d59b9-kk99j\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:36.117900 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.117887    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/529b10fb-7718-4c71-a2e4-44358d648c2d-tls-certs\") pod \"sub-test-model-108dbb-kserve-c89d59b9-kk99j\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:36.127980 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.127954    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfrc4\" (UniqueName: \"kubernetes.io/projected/529b10fb-7718-4c71-a2e4-44358d648c2d-kube-api-access-pfrc4\") pod \"sub-test-model-108dbb-kserve-c89d59b9-kk99j\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:36.221531 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.221492    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:36.499898 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.499864    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"]
Sep 10 16:05:36.501485 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:05:36.501452    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod529b10fb_7718_4c71_a2e4_44358d648c2d.slice/crio-a21383f9b361c15c089485219e726f1bce2cfebb1cd2eab3daaf9e2836faa87d WatchSource:0}: Error finding container a21383f9b361c15c089485219e726f1bce2cfebb1cd2eab3daaf9e2836faa87d: Status 404 returned error can't find the container with id a21383f9b361c15c089485219e726f1bce2cfebb1cd2eab3daaf9e2836faa87d
Sep 10 16:05:36.874011 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.873918    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j" event={"ID":"529b10fb-7718-4c71-a2e4-44358d648c2d","Type":"ContainerStarted","Data":"8b9b16836dabea3387f02b08d529c12bf6e9652f686b80f495550e8b4eae9cda"}
Sep 10 16:05:36.874011 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.873966    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j" event={"ID":"529b10fb-7718-4c71-a2e4-44358d648c2d","Type":"ContainerStarted","Data":"a21383f9b361c15c089485219e726f1bce2cfebb1cd2eab3daaf9e2836faa87d"}
Sep 10 16:05:36.892833 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:36.892785    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j" podStartSLOduration=1.892769265 podStartE2EDuration="1.892769265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:05:36.892390954 +0000 UTC m=+1415.696710218" watchObservedRunningTime="2026-09-10 16:05:36.892769265 +0000 UTC m=+1415.697088525"
Sep 10 16:05:46.222627 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:46.222573    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:46.223083 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:46.222643    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:46.240901 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:46.240867    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:46.957414 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:46.957384    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:05:47.955077 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:47.955044    2564 generic.go:356] "Generic (PLEG): container finished" podID="2d171cbb-6111-4067-b134-b9016a99b03f" containerID="07128b69afe2e5f8657d61d2c622c6dfe7ae6ebcd15ccf0f9f09c63ebc15f773" exitCode=0
Sep 10 16:05:47.955532 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:47.955128    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr" event={"ID":"2d171cbb-6111-4067-b134-b9016a99b03f","Type":"ContainerDied","Data":"07128b69afe2e5f8657d61d2c622c6dfe7ae6ebcd15ccf0f9f09c63ebc15f773"}
Sep 10 16:05:47.955635 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:47.955621    2564 scope.go:117] "RemoveContainer" containerID="07128b69afe2e5f8657d61d2c622c6dfe7ae6ebcd15ccf0f9f09c63ebc15f773"
Sep 10 16:05:47.956910 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:47.956883    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:05:47.956978 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:47.956955    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:05:47.957018 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:47.956987    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:05:48.963832 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:48.963799    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr" event={"ID":"2d171cbb-6111-4067-b134-b9016a99b03f","Type":"ContainerStarted","Data":"f31f4b2673fc534c1efff095362533fce92d40de3923fd80df8898f92beac8a9"}
Sep 10 16:05:49.643627 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:49.643590    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:49.643870 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:49.643850    2564 patch_prober.go:28] interesting pod/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 10 16:05:49.643926 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:49.643900    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr" podUID="2d171cbb-6111-4067-b134-b9016a99b03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 10 16:05:50.643795 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:50.643760    2564 patch_prober.go:28] interesting pod/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 10 16:05:50.644210 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:50.643813    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr" podUID="2d171cbb-6111-4067-b134-b9016a99b03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 10 16:05:51.644424 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:51.644373    2564 patch_prober.go:28] interesting pod/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 10 16:05:51.644839 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:51.644469    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr" podUID="2d171cbb-6111-4067-b134-b9016a99b03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 10 16:05:52.643858 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:52.643778    2564 patch_prober.go:28] interesting pod/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 10 16:05:52.643858 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:52.643833    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr" podUID="2d171cbb-6111-4067-b134-b9016a99b03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 10 16:05:53.644035 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:53.643996    2564 patch_prober.go:28] interesting pod/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 10 16:05:53.644462 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:53.644056    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr" podUID="2d171cbb-6111-4067-b134-b9016a99b03f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 10 16:05:53.707799 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:53.707737    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea0f68-mc844"]
Sep 10 16:05:53.712180 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:53.712128    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea0f68-mc844"
Sep 10 16:05:53.714828 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:53.714803    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2wghg\""
Sep 10 16:05:53.722706 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:53.722675    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea0f68-mc844"]
Sep 10 16:05:53.829782 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:53.829740    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e8ed47-c66c-450c-9178-cad4d5843ba1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ea0f68-mc844\" (UID: \"21e8ed47-c66c-450c-9178-cad4d5843ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea0f68-mc844"
Sep 10 16:05:53.931036 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:53.930993    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e8ed47-c66c-450c-9178-cad4d5843ba1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ea0f68-mc844\" (UID: \"21e8ed47-c66c-450c-9178-cad4d5843ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea0f68-mc844"
Sep 10 16:05:53.931940 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:53.931912    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e8ed47-c66c-450c-9178-cad4d5843ba1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ea0f68-mc844\" (UID: \"21e8ed47-c66c-450c-9178-cad4d5843ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea0f68-mc844"
Sep 10 16:05:54.028977 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:54.028938    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea0f68-mc844"
Sep 10 16:05:54.315146 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:54.315059    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea0f68-mc844"]
Sep 10 16:05:54.318919 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:05:54.318890    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e8ed47_c66c_450c_9178_cad4d5843ba1.slice/crio-dc9d521a212e86c0b35bbb78ef25268e0848c76b4bf5560ef2649b116d806eb4 WatchSource:0}: Error finding container dc9d521a212e86c0b35bbb78ef25268e0848c76b4bf5560ef2649b116d806eb4: Status 404 returned error can't find the container with id dc9d521a212e86c0b35bbb78ef25268e0848c76b4bf5560ef2649b116d806eb4
Sep 10 16:05:54.647663 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:54.647633    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:54.648062 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:54.648043    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:54.648784 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:54.648762    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:05:55.001939 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:55.001901    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea0f68-mc844" event={"ID":"21e8ed47-c66c-450c-9178-cad4d5843ba1","Type":"ContainerStarted","Data":"8dd96e721e2b11fbdcbde09ee3ef9b3cbf9bd1047cf5a592494a974fdf692b54"}
Sep 10 16:05:55.002184 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:55.001965    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea0f68-mc844" event={"ID":"21e8ed47-c66c-450c-9178-cad4d5843ba1","Type":"ContainerStarted","Data":"dc9d521a212e86c0b35bbb78ef25268e0848c76b4bf5560ef2649b116d806eb4"}
Sep 10 16:05:55.019265 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:55.019208    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea0f68-mc844" podStartSLOduration=2.01919177 podStartE2EDuration="2.01919177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:05:55.01861791 +0000 UTC m=+1433.822937173" watchObservedRunningTime="2026-09-10 16:05:55.01919177 +0000 UTC m=+1433.823511031"
Sep 10 16:05:56.009921 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:56.009889    2564 generic.go:356] "Generic (PLEG): container finished" podID="21e8ed47-c66c-450c-9178-cad4d5843ba1" containerID="8dd96e721e2b11fbdcbde09ee3ef9b3cbf9bd1047cf5a592494a974fdf692b54" exitCode=0
Sep 10 16:05:56.010448 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:56.009975    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea0f68-mc844" event={"ID":"21e8ed47-c66c-450c-9178-cad4d5843ba1","Type":"ContainerDied","Data":"8dd96e721e2b11fbdcbde09ee3ef9b3cbf9bd1047cf5a592494a974fdf692b54"}
Sep 10 16:05:57.162360 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:57.162334    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea0f68-mc844"
Sep 10 16:05:57.272801 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:57.272713    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e8ed47-c66c-450c-9178-cad4d5843ba1-maas-api-service-ca\") pod \"21e8ed47-c66c-450c-9178-cad4d5843ba1\" (UID: \"21e8ed47-c66c-450c-9178-cad4d5843ba1\") "
Sep 10 16:05:57.273122 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:57.273099    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e8ed47-c66c-450c-9178-cad4d5843ba1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "21e8ed47-c66c-450c-9178-cad4d5843ba1" (UID: "21e8ed47-c66c-450c-9178-cad4d5843ba1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:05:57.373812 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:57.373777    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e8ed47-c66c-450c-9178-cad4d5843ba1-maas-api-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:05:58.028995 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:58.028902    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea0f68-mc844"
Sep 10 16:05:58.028995 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:58.028922    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea0f68-mc844" event={"ID":"21e8ed47-c66c-450c-9178-cad4d5843ba1","Type":"ContainerDied","Data":"dc9d521a212e86c0b35bbb78ef25268e0848c76b4bf5560ef2649b116d806eb4"}
Sep 10 16:05:58.028995 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:05:58.028985    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc9d521a212e86c0b35bbb78ef25268e0848c76b4bf5560ef2649b116d806eb4"
Sep 10 16:06:02.609500 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:02.609460    2564 scope.go:117] "RemoveContainer" containerID="1441ffe072a500cd520ea203aed805636d00325b600ee4fd6904552645200921"
Sep 10 16:06:02.626748 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:02.626727    2564 scope.go:117] "RemoveContainer" containerID="9ffb4fb4d90b9cb8fe799eca00aba7502142e04794cf811296a23f64ebce8b3e"
Sep 10 16:06:02.634932 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:02.634914    2564 scope.go:117] "RemoveContainer" containerID="0ab9d40477235c7aef6f3bb0bf76dc58df35ab85548ccf7c0a62dd3690e0c827"
Sep 10 16:06:10.478116 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:10.478072    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea0f68-mc844"]
Sep 10 16:06:10.484591 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:10.484555    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ea0f68-mc844"]
Sep 10 16:06:11.673599 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:11.673566    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21e8ed47-c66c-450c-9178-cad4d5843ba1" path="/var/lib/kubelet/pods/21e8ed47-c66c-450c-9178-cad4d5843ba1/volumes"
Sep 10 16:06:19.013704 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:19.013672    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"]
Sep 10 16:06:19.014305 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:19.013952    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj" podUID="c0531a19-b416-414c-aea2-430cc11dbb23" containerName="istio-proxy" containerID="cri-o://c826fdef974e7549066f9854a0869facdd6305086186a470ca6e7a541dd7dd4c" gracePeriod=30
Sep 10 16:06:19.376367 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:19.376283    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ab381-5pbz6"]
Sep 10 16:06:19.376758 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:19.376744    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21e8ed47-c66c-450c-9178-cad4d5843ba1" containerName="revoke-keys"
Sep 10 16:06:19.376816 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:19.376760    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="21e8ed47-c66c-450c-9178-cad4d5843ba1" containerName="revoke-keys"
Sep 10 16:06:19.376867 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:19.376857    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="21e8ed47-c66c-450c-9178-cad4d5843ba1" containerName="revoke-keys"
Sep 10 16:06:19.381910 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:19.381891    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ab381-5pbz6"
Sep 10 16:06:19.384645 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:19.384626    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2wghg\""
Sep 10 16:06:19.393611 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:19.393588    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ab381-5pbz6"]
Sep 10 16:06:19.409589 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:19.409550    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0500293-aca2-48c8-a7d5-645b7b9790c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ab381-5pbz6\" (UID: \"a0500293-aca2-48c8-a7d5-645b7b9790c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ab381-5pbz6"
Sep 10 16:06:19.510867 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:19.510825    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0500293-aca2-48c8-a7d5-645b7b9790c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ab381-5pbz6\" (UID: \"a0500293-aca2-48c8-a7d5-645b7b9790c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ab381-5pbz6"
Sep 10 16:06:19.511516 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:19.511497    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0500293-aca2-48c8-a7d5-645b7b9790c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ab381-5pbz6\" (UID: \"a0500293-aca2-48c8-a7d5-645b7b9790c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ab381-5pbz6"
Sep 10 16:06:19.693699 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:19.693670    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ab381-5pbz6"
Sep 10 16:06:19.987327 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:19.987291    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ab381-5pbz6"]
Sep 10 16:06:19.988385 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:06:19.988345    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0500293_aca2_48c8_a7d5_645b7b9790c0.slice/crio-16b77d20ac225a28059e74d5dfabb4b74203a7d79360f5f198b1a85b3b251a91 WatchSource:0}: Error finding container 16b77d20ac225a28059e74d5dfabb4b74203a7d79360f5f198b1a85b3b251a91: Status 404 returned error can't find the container with id 16b77d20ac225a28059e74d5dfabb4b74203a7d79360f5f198b1a85b3b251a91
Sep 10 16:06:20.168458 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:20.168421    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ab381-5pbz6" event={"ID":"a0500293-aca2-48c8-a7d5-645b7b9790c0","Type":"ContainerStarted","Data":"16b77d20ac225a28059e74d5dfabb4b74203a7d79360f5f198b1a85b3b251a91"}
Sep 10 16:06:21.115281 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:21.115201    2564 patch_prober.go:28] interesting pod/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 16:06:21.115281 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:21.115257    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj" podUID="c0531a19-b416-414c-aea2-430cc11dbb23" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 16:06:21.176883 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:21.176851    2564 generic.go:356] "Generic (PLEG): container finished" podID="a0500293-aca2-48c8-a7d5-645b7b9790c0" containerID="5756186284b14d8d7cabd0e519d6388f971fb2c1e2cbbde3019debcdf9c56fd9" exitCode=0
Sep 10 16:06:21.177304 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:21.176938    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ab381-5pbz6" event={"ID":"a0500293-aca2-48c8-a7d5-645b7b9790c0","Type":"ContainerDied","Data":"5756186284b14d8d7cabd0e519d6388f971fb2c1e2cbbde3019debcdf9c56fd9"}
Sep 10 16:06:22.327090 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:22.327066    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ab381-5pbz6"
Sep 10 16:06:22.342374 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:22.342346    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0500293-aca2-48c8-a7d5-645b7b9790c0-maas-api-service-ca\") pod \"a0500293-aca2-48c8-a7d5-645b7b9790c0\" (UID: \"a0500293-aca2-48c8-a7d5-645b7b9790c0\") "
Sep 10 16:06:22.342680 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:22.342659    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0500293-aca2-48c8-a7d5-645b7b9790c0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a0500293-aca2-48c8-a7d5-645b7b9790c0" (UID: "a0500293-aca2-48c8-a7d5-645b7b9790c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:06:22.444210 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:22.444095    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0500293-aca2-48c8-a7d5-645b7b9790c0-maas-api-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:23.192725 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:23.192699    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ab381-5pbz6"
Sep 10 16:06:23.192907 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:23.192702    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ab381-5pbz6" event={"ID":"a0500293-aca2-48c8-a7d5-645b7b9790c0","Type":"ContainerDied","Data":"16b77d20ac225a28059e74d5dfabb4b74203a7d79360f5f198b1a85b3b251a91"}
Sep 10 16:06:23.192907 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:23.192818    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16b77d20ac225a28059e74d5dfabb4b74203a7d79360f5f198b1a85b3b251a91"
Sep 10 16:06:24.203203 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.203010    2564 generic.go:356] "Generic (PLEG): container finished" podID="c0531a19-b416-414c-aea2-430cc11dbb23" containerID="c826fdef974e7549066f9854a0869facdd6305086186a470ca6e7a541dd7dd4c" exitCode=0
Sep 10 16:06:24.203203 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.203111    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj" event={"ID":"c0531a19-b416-414c-aea2-430cc11dbb23","Type":"ContainerDied","Data":"c826fdef974e7549066f9854a0869facdd6305086186a470ca6e7a541dd7dd4c"}
Sep 10 16:06:24.203716 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.203188    2564 scope.go:117] "RemoveContainer" containerID="005024856a8b83fa662904f051c8c5104b14a5c3ea75164e5313f977df96050c"
Sep 10 16:06:24.285532 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.285465    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:06:24.364744 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.364713    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0531a19-b416-414c-aea2-430cc11dbb23-istio-podinfo\") pod \"c0531a19-b416-414c-aea2-430cc11dbb23\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") "
Sep 10 16:06:24.364931 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.364761    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-istio-data\") pod \"c0531a19-b416-414c-aea2-430cc11dbb23\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") "
Sep 10 16:06:24.364931 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.364801    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-workload-certs\") pod \"c0531a19-b416-414c-aea2-430cc11dbb23\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") "
Sep 10 16:06:24.364931 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.364862    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-credential-socket\") pod \"c0531a19-b416-414c-aea2-430cc11dbb23\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") "
Sep 10 16:06:24.364931 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.364895    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sfhj\" (UniqueName: \"kubernetes.io/projected/c0531a19-b416-414c-aea2-430cc11dbb23-kube-api-access-6sfhj\") pod \"c0531a19-b416-414c-aea2-430cc11dbb23\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") "
Sep 10 16:06:24.364931 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.364919    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0531a19-b416-414c-aea2-430cc11dbb23-istiod-ca-cert\") pod \"c0531a19-b416-414c-aea2-430cc11dbb23\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") "
Sep 10 16:06:24.365230 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.365014    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-istio-envoy\") pod \"c0531a19-b416-414c-aea2-430cc11dbb23\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") "
Sep 10 16:06:24.365230 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.365015    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-istio-data" (OuterVolumeSpecName: "istio-data") pod "c0531a19-b416-414c-aea2-430cc11dbb23" (UID: "c0531a19-b416-414c-aea2-430cc11dbb23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:24.365230 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.365041    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-workload-socket\") pod \"c0531a19-b416-414c-aea2-430cc11dbb23\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") "
Sep 10 16:06:24.365230 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.365071    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0531a19-b416-414c-aea2-430cc11dbb23-istio-token\") pod \"c0531a19-b416-414c-aea2-430cc11dbb23\" (UID: \"c0531a19-b416-414c-aea2-430cc11dbb23\") "
Sep 10 16:06:24.365726 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.365256    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c0531a19-b416-414c-aea2-430cc11dbb23" (UID: "c0531a19-b416-414c-aea2-430cc11dbb23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:24.365726 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.365264    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c0531a19-b416-414c-aea2-430cc11dbb23" (UID: "c0531a19-b416-414c-aea2-430cc11dbb23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:24.365726 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.365402    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0531a19-b416-414c-aea2-430cc11dbb23-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c0531a19-b416-414c-aea2-430cc11dbb23" (UID: "c0531a19-b416-414c-aea2-430cc11dbb23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:06:24.365726 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.365512    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c0531a19-b416-414c-aea2-430cc11dbb23" (UID: "c0531a19-b416-414c-aea2-430cc11dbb23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:24.365726 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.365592    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-istio-data\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:24.365726 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.365613    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-workload-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:24.365726 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.365628    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-credential-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:24.365726 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.365650    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0531a19-b416-414c-aea2-430cc11dbb23-istiod-ca-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:24.367579 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.367553    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0531a19-b416-414c-aea2-430cc11dbb23-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c0531a19-b416-414c-aea2-430cc11dbb23" (UID: "c0531a19-b416-414c-aea2-430cc11dbb23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 16:06:24.367855 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.367824    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c0531a19-b416-414c-aea2-430cc11dbb23" (UID: "c0531a19-b416-414c-aea2-430cc11dbb23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:24.367995 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.367970    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0531a19-b416-414c-aea2-430cc11dbb23-kube-api-access-6sfhj" (OuterVolumeSpecName: "kube-api-access-6sfhj") pod "c0531a19-b416-414c-aea2-430cc11dbb23" (UID: "c0531a19-b416-414c-aea2-430cc11dbb23"). InnerVolumeSpecName "kube-api-access-6sfhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:06:24.367995 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.367983    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0531a19-b416-414c-aea2-430cc11dbb23-istio-token" (OuterVolumeSpecName: "istio-token") pod "c0531a19-b416-414c-aea2-430cc11dbb23" (UID: "c0531a19-b416-414c-aea2-430cc11dbb23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:06:24.466409 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.466370    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0531a19-b416-414c-aea2-430cc11dbb23-istio-podinfo\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:24.466409 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.466403    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sfhj\" (UniqueName: \"kubernetes.io/projected/c0531a19-b416-414c-aea2-430cc11dbb23-kube-api-access-6sfhj\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:24.466409 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.466412    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-istio-envoy\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:24.466409 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.466421    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0531a19-b416-414c-aea2-430cc11dbb23-workload-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:24.466691 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:24.466430    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0531a19-b416-414c-aea2-430cc11dbb23-istio-token\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:25.212311 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:25.212274    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj" event={"ID":"c0531a19-b416-414c-aea2-430cc11dbb23","Type":"ContainerDied","Data":"14f130177028d70f1ef878dc551c92bb2d9eccdfe8639af7bb619d82c32dc032"}
Sep 10 16:06:25.212751 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:25.212339    2564 scope.go:117] "RemoveContainer" containerID="c826fdef974e7549066f9854a0869facdd6305086186a470ca6e7a541dd7dd4c"
Sep 10 16:06:25.212751 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:25.212288    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"
Sep 10 16:06:25.243899 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:25.243873    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"]
Sep 10 16:06:25.252217 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:25.252192    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ea0f68-openshift-default-69747cf87b-xgpbj"]
Sep 10 16:06:25.673860 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:25.673830    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0531a19-b416-414c-aea2-430cc11dbb23" path="/var/lib/kubelet/pods/c0531a19-b416-414c-aea2-430cc11dbb23/volumes"
Sep 10 16:06:35.560353 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:35.560311    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ab381-5pbz6"]
Sep 10 16:06:35.562542 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:35.562515    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ab381-5pbz6"]
Sep 10 16:06:35.678297 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:35.678261    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0500293-aca2-48c8-a7d5-645b7b9790c0" path="/var/lib/kubelet/pods/a0500293-aca2-48c8-a7d5-645b7b9790c0/volumes"
Sep 10 16:06:42.970503 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:42.970471    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"]
Sep 10 16:06:42.971036 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:42.970735    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr" podUID="2d171cbb-6111-4067-b134-b9016a99b03f" containerName="istio-proxy" containerID="cri-o://f31f4b2673fc534c1efff095362533fce92d40de3923fd80df8898f92beac8a9" gracePeriod=30
Sep 10 16:06:43.300594 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:43.300501    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-247f70-b57rk"]
Sep 10 16:06:43.301568 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:43.301534    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0531a19-b416-414c-aea2-430cc11dbb23" containerName="istio-proxy"
Sep 10 16:06:43.301568 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:43.301561    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0531a19-b416-414c-aea2-430cc11dbb23" containerName="istio-proxy"
Sep 10 16:06:43.301780 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:43.301594    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0500293-aca2-48c8-a7d5-645b7b9790c0" containerName="revoke-keys"
Sep 10 16:06:43.301780 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:43.301603    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0500293-aca2-48c8-a7d5-645b7b9790c0" containerName="revoke-keys"
Sep 10 16:06:43.301780 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:43.301729    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0531a19-b416-414c-aea2-430cc11dbb23" containerName="istio-proxy"
Sep 10 16:06:43.301780 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:43.301740    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0500293-aca2-48c8-a7d5-645b7b9790c0" containerName="revoke-keys"
Sep 10 16:06:43.305909 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:43.305887    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-247f70-b57rk"
Sep 10 16:06:43.309742 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:43.309709    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2wghg\""
Sep 10 16:06:43.313647 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:43.313608    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-247f70-b57rk"]
Sep 10 16:06:43.469730 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:43.469693    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/413cb543-87f2-4769-9617-9c44a3e28f5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-247f70-b57rk\" (UID: \"413cb543-87f2-4769-9617-9c44a3e28f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-247f70-b57rk"
Sep 10 16:06:43.571258 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:43.571142    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/413cb543-87f2-4769-9617-9c44a3e28f5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-247f70-b57rk\" (UID: \"413cb543-87f2-4769-9617-9c44a3e28f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-247f70-b57rk"
Sep 10 16:06:43.571757 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:43.571739    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/413cb543-87f2-4769-9617-9c44a3e28f5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-247f70-b57rk\" (UID: \"413cb543-87f2-4769-9617-9c44a3e28f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-247f70-b57rk"
Sep 10 16:06:43.630684 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:43.630654    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-247f70-b57rk"
Sep 10 16:06:43.910215 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:43.910115    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"]
Sep 10 16:06:43.917201 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:43.917173    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-247f70-b57rk"]
Sep 10 16:06:43.918942 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:06:43.918914    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413cb543_87f2_4769_9617_9c44a3e28f5e.slice/crio-760af97775b32b2c8b159eeb8db4e5fb3318f0db1a28db20b5615852f48b5102 WatchSource:0}: Error finding container 760af97775b32b2c8b159eeb8db4e5fb3318f0db1a28db20b5615852f48b5102: Status 404 returned error can't find the container with id 760af97775b32b2c8b159eeb8db4e5fb3318f0db1a28db20b5615852f48b5102
Sep 10 16:06:44.346360 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:44.346329    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-247f70-b57rk" event={"ID":"413cb543-87f2-4769-9617-9c44a3e28f5e","Type":"ContainerStarted","Data":"92784fecd04efce2bd51da36aacd32428393fef66c42ca7314eba77fa29688ba"}
Sep 10 16:06:44.346856 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:44.346388    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-247f70-b57rk" event={"ID":"413cb543-87f2-4769-9617-9c44a3e28f5e","Type":"ContainerStarted","Data":"760af97775b32b2c8b159eeb8db4e5fb3318f0db1a28db20b5615852f48b5102"}
Sep 10 16:06:44.366931 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:44.366853    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-247f70-b57rk" podStartSLOduration=1.3668369949999999 podStartE2EDuration="1.366836995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:06:44.36423555 +0000 UTC m=+1483.168554815" watchObservedRunningTime="2026-09-10 16:06:44.366836995 +0000 UTC m=+1483.171156256"
Sep 10 16:06:44.945933 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:44.945899    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"]
Sep 10 16:06:45.354091 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:45.354051    2564 generic.go:356] "Generic (PLEG): container finished" podID="413cb543-87f2-4769-9617-9c44a3e28f5e" containerID="92784fecd04efce2bd51da36aacd32428393fef66c42ca7314eba77fa29688ba" exitCode=0
Sep 10 16:06:45.354554 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:45.354138    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-247f70-b57rk" event={"ID":"413cb543-87f2-4769-9617-9c44a3e28f5e","Type":"ContainerDied","Data":"92784fecd04efce2bd51da36aacd32428393fef66c42ca7314eba77fa29688ba"}
Sep 10 16:06:46.514911 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:46.514884    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-247f70-b57rk"
Sep 10 16:06:46.606120 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:46.606021    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/413cb543-87f2-4769-9617-9c44a3e28f5e-maas-api-service-ca\") pod \"413cb543-87f2-4769-9617-9c44a3e28f5e\" (UID: \"413cb543-87f2-4769-9617-9c44a3e28f5e\") "
Sep 10 16:06:46.606378 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:46.606356    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/413cb543-87f2-4769-9617-9c44a3e28f5e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "413cb543-87f2-4769-9617-9c44a3e28f5e" (UID: "413cb543-87f2-4769-9617-9c44a3e28f5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:06:46.606605 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:46.606586    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/413cb543-87f2-4769-9617-9c44a3e28f5e-maas-api-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:47.370297 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:47.370266    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-247f70-b57rk"
Sep 10 16:06:47.370468 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:47.370308    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-247f70-b57rk" event={"ID":"413cb543-87f2-4769-9617-9c44a3e28f5e","Type":"ContainerDied","Data":"760af97775b32b2c8b159eeb8db4e5fb3318f0db1a28db20b5615852f48b5102"}
Sep 10 16:06:47.370468 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:47.370373    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="760af97775b32b2c8b159eeb8db4e5fb3318f0db1a28db20b5615852f48b5102"
Sep 10 16:06:48.235484 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.235405    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:06:48.325766 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.325725    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-workload-socket\") pod \"2d171cbb-6111-4067-b134-b9016a99b03f\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") "
Sep 10 16:06:48.325918 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.325806    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d171cbb-6111-4067-b134-b9016a99b03f-istiod-ca-cert\") pod \"2d171cbb-6111-4067-b134-b9016a99b03f\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") "
Sep 10 16:06:48.325918 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.325870    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-istio-envoy\") pod \"2d171cbb-6111-4067-b134-b9016a99b03f\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") "
Sep 10 16:06:48.326029 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.325931    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-workload-certs\") pod \"2d171cbb-6111-4067-b134-b9016a99b03f\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") "
Sep 10 16:06:48.326029 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.325973    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-credential-socket\") pod \"2d171cbb-6111-4067-b134-b9016a99b03f\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") "
Sep 10 16:06:48.326129 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.326025    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxqzh\" (UniqueName: \"kubernetes.io/projected/2d171cbb-6111-4067-b134-b9016a99b03f-kube-api-access-vxqzh\") pod \"2d171cbb-6111-4067-b134-b9016a99b03f\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") "
Sep 10 16:06:48.326129 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.326046    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2d171cbb-6111-4067-b134-b9016a99b03f" (UID: "2d171cbb-6111-4067-b134-b9016a99b03f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:48.326129 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.326077    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d171cbb-6111-4067-b134-b9016a99b03f-istio-token\") pod \"2d171cbb-6111-4067-b134-b9016a99b03f\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") "
Sep 10 16:06:48.326335 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.326199    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-istio-data\") pod \"2d171cbb-6111-4067-b134-b9016a99b03f\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") "
Sep 10 16:06:48.326335 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.326235    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d171cbb-6111-4067-b134-b9016a99b03f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2d171cbb-6111-4067-b134-b9016a99b03f" (UID: "2d171cbb-6111-4067-b134-b9016a99b03f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:06:48.326335 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.326239    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d171cbb-6111-4067-b134-b9016a99b03f-istio-podinfo\") pod \"2d171cbb-6111-4067-b134-b9016a99b03f\" (UID: \"2d171cbb-6111-4067-b134-b9016a99b03f\") "
Sep 10 16:06:48.326498 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.326387    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2d171cbb-6111-4067-b134-b9016a99b03f" (UID: "2d171cbb-6111-4067-b134-b9016a99b03f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:48.326580 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.326559    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2d171cbb-6111-4067-b134-b9016a99b03f" (UID: "2d171cbb-6111-4067-b134-b9016a99b03f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:48.326636 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.326603    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-istio-data" (OuterVolumeSpecName: "istio-data") pod "2d171cbb-6111-4067-b134-b9016a99b03f" (UID: "2d171cbb-6111-4067-b134-b9016a99b03f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:48.326993 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.326971    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-istio-data\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:48.327045 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.326996    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-workload-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:48.327045 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.327012    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d171cbb-6111-4067-b134-b9016a99b03f-istiod-ca-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:48.327045 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.327025    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-workload-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:48.327045 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.327037    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-credential-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:48.328453 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.328423    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2d171cbb-6111-4067-b134-b9016a99b03f" (UID: "2d171cbb-6111-4067-b134-b9016a99b03f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:48.328453 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.328433    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d171cbb-6111-4067-b134-b9016a99b03f-istio-token" (OuterVolumeSpecName: "istio-token") pod "2d171cbb-6111-4067-b134-b9016a99b03f" (UID: "2d171cbb-6111-4067-b134-b9016a99b03f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:06:48.328607 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.328526    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d171cbb-6111-4067-b134-b9016a99b03f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2d171cbb-6111-4067-b134-b9016a99b03f" (UID: "2d171cbb-6111-4067-b134-b9016a99b03f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 16:06:48.328806 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.328784    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d171cbb-6111-4067-b134-b9016a99b03f-kube-api-access-vxqzh" (OuterVolumeSpecName: "kube-api-access-vxqzh") pod "2d171cbb-6111-4067-b134-b9016a99b03f" (UID: "2d171cbb-6111-4067-b134-b9016a99b03f"). InnerVolumeSpecName "kube-api-access-vxqzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:06:48.377526 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.377495    2564 generic.go:356] "Generic (PLEG): container finished" podID="2d171cbb-6111-4067-b134-b9016a99b03f" containerID="f31f4b2673fc534c1efff095362533fce92d40de3923fd80df8898f92beac8a9" exitCode=0
Sep 10 16:06:48.377737 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.377547    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"
Sep 10 16:06:48.377737 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.377581    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr" event={"ID":"2d171cbb-6111-4067-b134-b9016a99b03f","Type":"ContainerDied","Data":"f31f4b2673fc534c1efff095362533fce92d40de3923fd80df8898f92beac8a9"}
Sep 10 16:06:48.377737 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.377649    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr" event={"ID":"2d171cbb-6111-4067-b134-b9016a99b03f","Type":"ContainerDied","Data":"a714d98d70f074d0071e373df22f554225c50dd0eff16a590c278496cae50e98"}
Sep 10 16:06:48.377737 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.377680    2564 scope.go:117] "RemoveContainer" containerID="f31f4b2673fc534c1efff095362533fce92d40de3923fd80df8898f92beac8a9"
Sep 10 16:06:48.387869 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.387853    2564 scope.go:117] "RemoveContainer" containerID="07128b69afe2e5f8657d61d2c622c6dfe7ae6ebcd15ccf0f9f09c63ebc15f773"
Sep 10 16:06:48.396894 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.396872    2564 scope.go:117] "RemoveContainer" containerID="f31f4b2673fc534c1efff095362533fce92d40de3923fd80df8898f92beac8a9"
Sep 10 16:06:48.397265 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:06:48.397245    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f31f4b2673fc534c1efff095362533fce92d40de3923fd80df8898f92beac8a9\": container with ID starting with f31f4b2673fc534c1efff095362533fce92d40de3923fd80df8898f92beac8a9 not found: ID does not exist" containerID="f31f4b2673fc534c1efff095362533fce92d40de3923fd80df8898f92beac8a9"
Sep 10 16:06:48.397377 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.397274    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f31f4b2673fc534c1efff095362533fce92d40de3923fd80df8898f92beac8a9"} err="failed to get container status \"f31f4b2673fc534c1efff095362533fce92d40de3923fd80df8898f92beac8a9\": rpc error: code = NotFound desc = could not find container \"f31f4b2673fc534c1efff095362533fce92d40de3923fd80df8898f92beac8a9\": container with ID starting with f31f4b2673fc534c1efff095362533fce92d40de3923fd80df8898f92beac8a9 not found: ID does not exist"
Sep 10 16:06:48.397377 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.397291    2564 scope.go:117] "RemoveContainer" containerID="07128b69afe2e5f8657d61d2c622c6dfe7ae6ebcd15ccf0f9f09c63ebc15f773"
Sep 10 16:06:48.397594 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:06:48.397573    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07128b69afe2e5f8657d61d2c622c6dfe7ae6ebcd15ccf0f9f09c63ebc15f773\": container with ID starting with 07128b69afe2e5f8657d61d2c622c6dfe7ae6ebcd15ccf0f9f09c63ebc15f773 not found: ID does not exist" containerID="07128b69afe2e5f8657d61d2c622c6dfe7ae6ebcd15ccf0f9f09c63ebc15f773"
Sep 10 16:06:48.397665 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.397600    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07128b69afe2e5f8657d61d2c622c6dfe7ae6ebcd15ccf0f9f09c63ebc15f773"} err="failed to get container status \"07128b69afe2e5f8657d61d2c622c6dfe7ae6ebcd15ccf0f9f09c63ebc15f773\": rpc error: code = NotFound desc = could not find container \"07128b69afe2e5f8657d61d2c622c6dfe7ae6ebcd15ccf0f9f09c63ebc15f773\": container with ID starting with 07128b69afe2e5f8657d61d2c622c6dfe7ae6ebcd15ccf0f9f09c63ebc15f773 not found: ID does not exist"
Sep 10 16:06:48.406531 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.406506    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"]
Sep 10 16:06:48.412999 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.412969    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2ab381-openshift-default-5cd7f587b-l5hzr"]
Sep 10 16:06:48.428043 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.428010    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d171cbb-6111-4067-b134-b9016a99b03f-istio-envoy\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:48.428043 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.428040    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxqzh\" (UniqueName: \"kubernetes.io/projected/2d171cbb-6111-4067-b134-b9016a99b03f-kube-api-access-vxqzh\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:48.428239 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.428054    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d171cbb-6111-4067-b134-b9016a99b03f-istio-token\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:48.428239 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:48.428069    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d171cbb-6111-4067-b134-b9016a99b03f-istio-podinfo\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:49.674318 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:49.674288    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d171cbb-6111-4067-b134-b9016a99b03f" path="/var/lib/kubelet/pods/2d171cbb-6111-4067-b134-b9016a99b03f/volumes"
Sep 10 16:06:54.242989 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:54.242944    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-247f70-b57rk"]
Sep 10 16:06:54.246141 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:54.246115    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-247f70-b57rk"]
Sep 10 16:06:55.673113 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:55.673084    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="413cb543-87f2-4769-9617-9c44a3e28f5e" path="/var/lib/kubelet/pods/413cb543-87f2-4769-9617-9c44a3e28f5e/volumes"
Sep 10 16:06:58.932694 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:58.932647    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg" podUID="9014c790-cf33-4579-9eba-ae8b2bd0ddd3" containerName="main" containerID="cri-o://fe8226743c7303f74e47e3a91c7a3cd3de1a107f98ea366170f7262eab649029" gracePeriod=45
Sep 10 16:06:59.194550 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.194492    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:06:59.240835 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.240782    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-home\") pod \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") "
Sep 10 16:06:59.241035 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.240852    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zp6bp\" (UniqueName: \"kubernetes.io/projected/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-kube-api-access-zp6bp\") pod \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") "
Sep 10 16:06:59.241035 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.240884    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-dshm\") pod \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") "
Sep 10 16:06:59.241035 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.240910    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-model-cache\") pod \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") "
Sep 10 16:06:59.241035 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.240931    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-tmp-dir\") pod \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") "
Sep 10 16:06:59.241035 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.240985    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-tls-certs\") pod \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\" (UID: \"9014c790-cf33-4579-9eba-ae8b2bd0ddd3\") "
Sep 10 16:06:59.241301 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.241103    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-home" (OuterVolumeSpecName: "home") pod "9014c790-cf33-4579-9eba-ae8b2bd0ddd3" (UID: "9014c790-cf33-4579-9eba-ae8b2bd0ddd3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:59.241416 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.241390    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9014c790-cf33-4579-9eba-ae8b2bd0ddd3" (UID: "9014c790-cf33-4579-9eba-ae8b2bd0ddd3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:59.241480 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.241415    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-home\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:59.241525 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.241486    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-model-cache" (OuterVolumeSpecName: "model-cache") pod "9014c790-cf33-4579-9eba-ae8b2bd0ddd3" (UID: "9014c790-cf33-4579-9eba-ae8b2bd0ddd3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:59.243478 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.243449    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-kube-api-access-zp6bp" (OuterVolumeSpecName: "kube-api-access-zp6bp") pod "9014c790-cf33-4579-9eba-ae8b2bd0ddd3" (UID: "9014c790-cf33-4579-9eba-ae8b2bd0ddd3"). InnerVolumeSpecName "kube-api-access-zp6bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:06:59.243748 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.243722    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9014c790-cf33-4579-9eba-ae8b2bd0ddd3" (UID: "9014c790-cf33-4579-9eba-ae8b2bd0ddd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:06:59.243834 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.243816    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-dshm" (OuterVolumeSpecName: "dshm") pod "9014c790-cf33-4579-9eba-ae8b2bd0ddd3" (UID: "9014c790-cf33-4579-9eba-ae8b2bd0ddd3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:06:59.342304 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.342252    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp6bp\" (UniqueName: \"kubernetes.io/projected/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-kube-api-access-zp6bp\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:59.342304 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.342296    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-dshm\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:59.342304 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.342306    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-model-cache\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:59.342304 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.342315    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-tmp-dir\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:59.342304 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.342322    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9014c790-cf33-4579-9eba-ae8b2bd0ddd3-tls-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:06:59.446002 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.445906    2564 generic.go:356] "Generic (PLEG): container finished" podID="9014c790-cf33-4579-9eba-ae8b2bd0ddd3" containerID="fe8226743c7303f74e47e3a91c7a3cd3de1a107f98ea366170f7262eab649029" exitCode=0
Sep 10 16:06:59.446002 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.445959    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg" event={"ID":"9014c790-cf33-4579-9eba-ae8b2bd0ddd3","Type":"ContainerDied","Data":"fe8226743c7303f74e47e3a91c7a3cd3de1a107f98ea366170f7262eab649029"}
Sep 10 16:06:59.446002 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.445989    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg" event={"ID":"9014c790-cf33-4579-9eba-ae8b2bd0ddd3","Type":"ContainerDied","Data":"b31485c78f5137f1902135f74065713e72a1bd023b7dc431ecc0acdaf0fbc16f"}
Sep 10 16:06:59.446002 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.446005    2564 scope.go:117] "RemoveContainer" containerID="fe8226743c7303f74e47e3a91c7a3cd3de1a107f98ea366170f7262eab649029"
Sep 10 16:06:59.446328 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.445962    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"
Sep 10 16:06:59.455531 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.455513    2564 scope.go:117] "RemoveContainer" containerID="fe8226743c7303f74e47e3a91c7a3cd3de1a107f98ea366170f7262eab649029"
Sep 10 16:06:59.455780 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:06:59.455763    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe8226743c7303f74e47e3a91c7a3cd3de1a107f98ea366170f7262eab649029\": container with ID starting with fe8226743c7303f74e47e3a91c7a3cd3de1a107f98ea366170f7262eab649029 not found: ID does not exist" containerID="fe8226743c7303f74e47e3a91c7a3cd3de1a107f98ea366170f7262eab649029"
Sep 10 16:06:59.455828 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.455788    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe8226743c7303f74e47e3a91c7a3cd3de1a107f98ea366170f7262eab649029"} err="failed to get container status \"fe8226743c7303f74e47e3a91c7a3cd3de1a107f98ea366170f7262eab649029\": rpc error: code = NotFound desc = could not find container \"fe8226743c7303f74e47e3a91c7a3cd3de1a107f98ea366170f7262eab649029\": container with ID starting with fe8226743c7303f74e47e3a91c7a3cd3de1a107f98ea366170f7262eab649029 not found: ID does not exist"
Sep 10 16:06:59.470029 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.470002    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"]
Sep 10 16:06:59.474516 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.474493    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b52270/sub-test-model-b52270-kserve-666bdf5d4c-gkmjg"]
Sep 10 16:06:59.673810 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.673766    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9014c790-cf33-4579-9eba-ae8b2bd0ddd3" path="/var/lib/kubelet/pods/9014c790-cf33-4579-9eba-ae8b2bd0ddd3/volumes"
Sep 10 16:06:59.966556 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:06:59.966520    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j" podUID="529b10fb-7718-4c71-a2e4-44358d648c2d" containerName="main" containerID="cri-o://8b9b16836dabea3387f02b08d529c12bf6e9652f686b80f495550e8b4eae9cda" gracePeriod=45
Sep 10 16:07:00.228508 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.228451    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:07:00.250184 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.250128    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-tmp-dir\") pod \"529b10fb-7718-4c71-a2e4-44358d648c2d\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") "
Sep 10 16:07:00.250368 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.250246    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-model-cache\") pod \"529b10fb-7718-4c71-a2e4-44358d648c2d\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") "
Sep 10 16:07:00.250368 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.250292    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/529b10fb-7718-4c71-a2e4-44358d648c2d-tls-certs\") pod \"529b10fb-7718-4c71-a2e4-44358d648c2d\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") "
Sep 10 16:07:00.250368 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.250352    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-home\") pod \"529b10fb-7718-4c71-a2e4-44358d648c2d\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") "
Sep 10 16:07:00.250548 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.250425    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "529b10fb-7718-4c71-a2e4-44358d648c2d" (UID: "529b10fb-7718-4c71-a2e4-44358d648c2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:00.250548 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.250432    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfrc4\" (UniqueName: \"kubernetes.io/projected/529b10fb-7718-4c71-a2e4-44358d648c2d-kube-api-access-pfrc4\") pod \"529b10fb-7718-4c71-a2e4-44358d648c2d\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") "
Sep 10 16:07:00.250548 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.250502    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-dshm\") pod \"529b10fb-7718-4c71-a2e4-44358d648c2d\" (UID: \"529b10fb-7718-4c71-a2e4-44358d648c2d\") "
Sep 10 16:07:00.250917 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.250890    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-model-cache" (OuterVolumeSpecName: "model-cache") pod "529b10fb-7718-4c71-a2e4-44358d648c2d" (UID: "529b10fb-7718-4c71-a2e4-44358d648c2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:00.250917 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.250908    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-home" (OuterVolumeSpecName: "home") pod "529b10fb-7718-4c71-a2e4-44358d648c2d" (UID: "529b10fb-7718-4c71-a2e4-44358d648c2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:00.251060 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.250974    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-tmp-dir\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:00.253145 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.253113    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-dshm" (OuterVolumeSpecName: "dshm") pod "529b10fb-7718-4c71-a2e4-44358d648c2d" (UID: "529b10fb-7718-4c71-a2e4-44358d648c2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:00.253409 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.253388    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/529b10fb-7718-4c71-a2e4-44358d648c2d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "529b10fb-7718-4c71-a2e4-44358d648c2d" (UID: "529b10fb-7718-4c71-a2e4-44358d648c2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:07:00.253565 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.253541    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529b10fb-7718-4c71-a2e4-44358d648c2d-kube-api-access-pfrc4" (OuterVolumeSpecName: "kube-api-access-pfrc4") pod "529b10fb-7718-4c71-a2e4-44358d648c2d" (UID: "529b10fb-7718-4c71-a2e4-44358d648c2d"). InnerVolumeSpecName "kube-api-access-pfrc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:07:00.351835 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.351802    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfrc4\" (UniqueName: \"kubernetes.io/projected/529b10fb-7718-4c71-a2e4-44358d648c2d-kube-api-access-pfrc4\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:00.351835 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.351831    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-dshm\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:00.351835 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.351843    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-model-cache\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:00.352088 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.351851    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/529b10fb-7718-4c71-a2e4-44358d648c2d-tls-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:00.352088 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.351858    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/529b10fb-7718-4c71-a2e4-44358d648c2d-home\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:00.456656 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.456618    2564 generic.go:356] "Generic (PLEG): container finished" podID="529b10fb-7718-4c71-a2e4-44358d648c2d" containerID="8b9b16836dabea3387f02b08d529c12bf6e9652f686b80f495550e8b4eae9cda" exitCode=0
Sep 10 16:07:00.456863 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.456663    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"
Sep 10 16:07:00.456863 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.456708    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j" event={"ID":"529b10fb-7718-4c71-a2e4-44358d648c2d","Type":"ContainerDied","Data":"8b9b16836dabea3387f02b08d529c12bf6e9652f686b80f495550e8b4eae9cda"}
Sep 10 16:07:00.456863 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.456752    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j" event={"ID":"529b10fb-7718-4c71-a2e4-44358d648c2d","Type":"ContainerDied","Data":"a21383f9b361c15c089485219e726f1bce2cfebb1cd2eab3daaf9e2836faa87d"}
Sep 10 16:07:00.456863 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.456771    2564 scope.go:117] "RemoveContainer" containerID="8b9b16836dabea3387f02b08d529c12bf6e9652f686b80f495550e8b4eae9cda"
Sep 10 16:07:00.466947 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.466929    2564 scope.go:117] "RemoveContainer" containerID="8b9b16836dabea3387f02b08d529c12bf6e9652f686b80f495550e8b4eae9cda"
Sep 10 16:07:00.467302 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:07:00.467281    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b9b16836dabea3387f02b08d529c12bf6e9652f686b80f495550e8b4eae9cda\": container with ID starting with 8b9b16836dabea3387f02b08d529c12bf6e9652f686b80f495550e8b4eae9cda not found: ID does not exist" containerID="8b9b16836dabea3387f02b08d529c12bf6e9652f686b80f495550e8b4eae9cda"
Sep 10 16:07:00.467365 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.467315    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b9b16836dabea3387f02b08d529c12bf6e9652f686b80f495550e8b4eae9cda"} err="failed to get container status \"8b9b16836dabea3387f02b08d529c12bf6e9652f686b80f495550e8b4eae9cda\": rpc error: code = NotFound desc = could not find container \"8b9b16836dabea3387f02b08d529c12bf6e9652f686b80f495550e8b4eae9cda\": container with ID starting with 8b9b16836dabea3387f02b08d529c12bf6e9652f686b80f495550e8b4eae9cda not found: ID does not exist"
Sep 10 16:07:00.480822 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.480755    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"]
Sep 10 16:07:00.485121 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:00.485096    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-108dbb/sub-test-model-108dbb-kserve-c89d59b9-kk99j"]
Sep 10 16:07:01.359717 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.359684    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"]
Sep 10 16:07:01.360094 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.359965    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x" podUID="45e66617-3de1-49ce-9d8b-0bc626c24765" containerName="istio-proxy" containerID="cri-o://62a60136deb007512d06b421ceb8f7a922fbf1c71cb0753e1b962c53b4d47e70" gracePeriod=30
Sep 10 16:07:01.673647 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.673602    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="529b10fb-7718-4c71-a2e4-44358d648c2d" path="/var/lib/kubelet/pods/529b10fb-7718-4c71-a2e4-44358d648c2d/volumes"
Sep 10 16:07:01.713343 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.713311    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-298d9b-hl6rb"]
Sep 10 16:07:01.713776 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.713760    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0531a19-b416-414c-aea2-430cc11dbb23" containerName="istio-proxy"
Sep 10 16:07:01.713776 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.713777    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0531a19-b416-414c-aea2-430cc11dbb23" containerName="istio-proxy"
Sep 10 16:07:01.713886 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.713790    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d171cbb-6111-4067-b134-b9016a99b03f" containerName="istio-proxy"
Sep 10 16:07:01.713886 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.713796    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d171cbb-6111-4067-b134-b9016a99b03f" containerName="istio-proxy"
Sep 10 16:07:01.713886 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.713804    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="529b10fb-7718-4c71-a2e4-44358d648c2d" containerName="main"
Sep 10 16:07:01.713886 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.713809    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="529b10fb-7718-4c71-a2e4-44358d648c2d" containerName="main"
Sep 10 16:07:01.713886 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.713829    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="413cb543-87f2-4769-9617-9c44a3e28f5e" containerName="revoke-keys"
Sep 10 16:07:01.713886 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.713835    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="413cb543-87f2-4769-9617-9c44a3e28f5e" containerName="revoke-keys"
Sep 10 16:07:01.713886 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.713847    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9014c790-cf33-4579-9eba-ae8b2bd0ddd3" containerName="main"
Sep 10 16:07:01.713886 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.713851    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9014c790-cf33-4579-9eba-ae8b2bd0ddd3" containerName="main"
Sep 10 16:07:01.714254 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.713919    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d171cbb-6111-4067-b134-b9016a99b03f" containerName="istio-proxy"
Sep 10 16:07:01.714254 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.713926    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="529b10fb-7718-4c71-a2e4-44358d648c2d" containerName="main"
Sep 10 16:07:01.714254 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.713934    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0531a19-b416-414c-aea2-430cc11dbb23" containerName="istio-proxy"
Sep 10 16:07:01.714254 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.713940    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="413cb543-87f2-4769-9617-9c44a3e28f5e" containerName="revoke-keys"
Sep 10 16:07:01.714254 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.713947    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9014c790-cf33-4579-9eba-ae8b2bd0ddd3" containerName="main"
Sep 10 16:07:01.717132 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.717112    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-298d9b-hl6rb"
Sep 10 16:07:01.719714 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.719694    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2wghg\""
Sep 10 16:07:01.728911 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.728882    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-298d9b-hl6rb"]
Sep 10 16:07:01.768115 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.768085    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0cdc83a-1478-4db0-9541-7d515cec437c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-298d9b-hl6rb\" (UID: \"a0cdc83a-1478-4db0-9541-7d515cec437c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-298d9b-hl6rb"
Sep 10 16:07:01.868803 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.868764    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0cdc83a-1478-4db0-9541-7d515cec437c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-298d9b-hl6rb\" (UID: \"a0cdc83a-1478-4db0-9541-7d515cec437c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-298d9b-hl6rb"
Sep 10 16:07:01.869454 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:01.869433    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0cdc83a-1478-4db0-9541-7d515cec437c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-298d9b-hl6rb\" (UID: \"a0cdc83a-1478-4db0-9541-7d515cec437c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-298d9b-hl6rb"
Sep 10 16:07:02.030977 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:02.030881    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-298d9b-hl6rb"
Sep 10 16:07:02.320602 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:02.320579    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-298d9b-hl6rb"]
Sep 10 16:07:02.323206 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:07:02.323177    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0cdc83a_1478_4db0_9541_7d515cec437c.slice/crio-df9c011bc7ffd6dc8d812da56913e09d4eef5ea9b1a8bce4fc1b1a1fd8da3605 WatchSource:0}: Error finding container df9c011bc7ffd6dc8d812da56913e09d4eef5ea9b1a8bce4fc1b1a1fd8da3605: Status 404 returned error can't find the container with id df9c011bc7ffd6dc8d812da56913e09d4eef5ea9b1a8bce4fc1b1a1fd8da3605
Sep 10 16:07:02.450888 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:02.450862    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-snm4z_f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f/console-operator/1.log"
Sep 10 16:07:02.451938 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:02.451916    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-snm4z_f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f/console-operator/1.log"
Sep 10 16:07:02.482086 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:02.482055    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-298d9b-hl6rb" event={"ID":"a0cdc83a-1478-4db0-9541-7d515cec437c","Type":"ContainerStarted","Data":"df9c011bc7ffd6dc8d812da56913e09d4eef5ea9b1a8bce4fc1b1a1fd8da3605"}
Sep 10 16:07:03.490077 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:03.490036    2564 generic.go:356] "Generic (PLEG): container finished" podID="a0cdc83a-1478-4db0-9541-7d515cec437c" containerID="6198e53616a8f5d77157927855756c3243e57bb66c9f92e1dceb016f9332a229" exitCode=0
Sep 10 16:07:03.490507 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:03.490123    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-298d9b-hl6rb" event={"ID":"a0cdc83a-1478-4db0-9541-7d515cec437c","Type":"ContainerDied","Data":"6198e53616a8f5d77157927855756c3243e57bb66c9f92e1dceb016f9332a229"}
Sep 10 16:07:04.636411 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:04.636386    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-298d9b-hl6rb"
Sep 10 16:07:04.701171 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:04.701136    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0cdc83a-1478-4db0-9541-7d515cec437c-maas-api-service-ca\") pod \"a0cdc83a-1478-4db0-9541-7d515cec437c\" (UID: \"a0cdc83a-1478-4db0-9541-7d515cec437c\") "
Sep 10 16:07:04.701505 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:04.701484    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0cdc83a-1478-4db0-9541-7d515cec437c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a0cdc83a-1478-4db0-9541-7d515cec437c" (UID: "a0cdc83a-1478-4db0-9541-7d515cec437c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:07:04.802351 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:04.802267    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0cdc83a-1478-4db0-9541-7d515cec437c-maas-api-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:05.503801 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:05.503767    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-298d9b-hl6rb"
Sep 10 16:07:05.503983 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:05.503794    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-298d9b-hl6rb" event={"ID":"a0cdc83a-1478-4db0-9541-7d515cec437c","Type":"ContainerDied","Data":"df9c011bc7ffd6dc8d812da56913e09d4eef5ea9b1a8bce4fc1b1a1fd8da3605"}
Sep 10 16:07:05.503983 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:05.503873    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df9c011bc7ffd6dc8d812da56913e09d4eef5ea9b1a8bce4fc1b1a1fd8da3605"
Sep 10 16:07:06.512963 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.512932    2564 generic.go:356] "Generic (PLEG): container finished" podID="45e66617-3de1-49ce-9d8b-0bc626c24765" containerID="62a60136deb007512d06b421ceb8f7a922fbf1c71cb0753e1b962c53b4d47e70" exitCode=0
Sep 10 16:07:06.513556 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.513011    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x" event={"ID":"45e66617-3de1-49ce-9d8b-0bc626c24765","Type":"ContainerDied","Data":"62a60136deb007512d06b421ceb8f7a922fbf1c71cb0753e1b962c53b4d47e70"}
Sep 10 16:07:06.513556 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.513071    2564 scope.go:117] "RemoveContainer" containerID="79a5e3ea3873e7721a42f5c585e890a76e3c06b0585ae9855021b597550c29ad"
Sep 10 16:07:06.630243 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.630172    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:07:06.722644 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.722605    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-workload-certs\") pod \"45e66617-3de1-49ce-9d8b-0bc626c24765\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") "
Sep 10 16:07:06.722644 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.722648    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e66617-3de1-49ce-9d8b-0bc626c24765-istiod-ca-cert\") pod \"45e66617-3de1-49ce-9d8b-0bc626c24765\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") "
Sep 10 16:07:06.722906 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.722679    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-credential-socket\") pod \"45e66617-3de1-49ce-9d8b-0bc626c24765\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") "
Sep 10 16:07:06.722906 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.722719    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-workload-socket\") pod \"45e66617-3de1-49ce-9d8b-0bc626c24765\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") "
Sep 10 16:07:06.722906 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.722757    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-envoy\") pod \"45e66617-3de1-49ce-9d8b-0bc626c24765\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") "
Sep 10 16:07:06.722906 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.722826    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mx95h\" (UniqueName: \"kubernetes.io/projected/45e66617-3de1-49ce-9d8b-0bc626c24765-kube-api-access-mx95h\") pod \"45e66617-3de1-49ce-9d8b-0bc626c24765\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") "
Sep 10 16:07:06.722906 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.722877    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-data\") pod \"45e66617-3de1-49ce-9d8b-0bc626c24765\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") "
Sep 10 16:07:06.722906 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.722897    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-token\") pod \"45e66617-3de1-49ce-9d8b-0bc626c24765\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") "
Sep 10 16:07:06.723255 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.722915    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "45e66617-3de1-49ce-9d8b-0bc626c24765" (UID: "45e66617-3de1-49ce-9d8b-0bc626c24765"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:06.723255 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.722935    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-podinfo\") pod \"45e66617-3de1-49ce-9d8b-0bc626c24765\" (UID: \"45e66617-3de1-49ce-9d8b-0bc626c24765\") "
Sep 10 16:07:06.723255 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.723107    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-data" (OuterVolumeSpecName: "istio-data") pod "45e66617-3de1-49ce-9d8b-0bc626c24765" (UID: "45e66617-3de1-49ce-9d8b-0bc626c24765"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:06.723255 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.723199    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e66617-3de1-49ce-9d8b-0bc626c24765-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "45e66617-3de1-49ce-9d8b-0bc626c24765" (UID: "45e66617-3de1-49ce-9d8b-0bc626c24765"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:07:06.723255 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.723247    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "45e66617-3de1-49ce-9d8b-0bc626c24765" (UID: "45e66617-3de1-49ce-9d8b-0bc626c24765"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:06.723454 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.723273    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "45e66617-3de1-49ce-9d8b-0bc626c24765" (UID: "45e66617-3de1-49ce-9d8b-0bc626c24765"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:06.723454 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.723406    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-workload-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:06.723454 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.723425    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e66617-3de1-49ce-9d8b-0bc626c24765-istiod-ca-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:06.723454 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.723439    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-credential-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:06.723454 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.723451    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-workload-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:06.723625 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.723465    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-data\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:06.725186 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.725144    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "45e66617-3de1-49ce-9d8b-0bc626c24765" (UID: "45e66617-3de1-49ce-9d8b-0bc626c24765"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:06.725298 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.725227    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e66617-3de1-49ce-9d8b-0bc626c24765-kube-api-access-mx95h" (OuterVolumeSpecName: "kube-api-access-mx95h") pod "45e66617-3de1-49ce-9d8b-0bc626c24765" (UID: "45e66617-3de1-49ce-9d8b-0bc626c24765"). InnerVolumeSpecName "kube-api-access-mx95h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:07:06.725674 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.725653    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "45e66617-3de1-49ce-9d8b-0bc626c24765" (UID: "45e66617-3de1-49ce-9d8b-0bc626c24765"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 16:07:06.725743 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.725655    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-token" (OuterVolumeSpecName: "istio-token") pod "45e66617-3de1-49ce-9d8b-0bc626c24765" (UID: "45e66617-3de1-49ce-9d8b-0bc626c24765"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:07:06.824555 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.824521    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-envoy\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:06.824555 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.824551    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx95h\" (UniqueName: \"kubernetes.io/projected/45e66617-3de1-49ce-9d8b-0bc626c24765-kube-api-access-mx95h\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:06.824555 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.824562    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-token\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:06.824796 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:06.824571    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e66617-3de1-49ce-9d8b-0bc626c24765-istio-podinfo\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:07.521382 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:07.521357    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"
Sep 10 16:07:07.521382 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:07.521376    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x" event={"ID":"45e66617-3de1-49ce-9d8b-0bc626c24765","Type":"ContainerDied","Data":"c796c2cbb22f5cb3b0ff00ff89b6969cb797c77e182acca995654c999f4ef32c"}
Sep 10 16:07:07.521849 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:07.521440    2564 scope.go:117] "RemoveContainer" containerID="62a60136deb007512d06b421ceb8f7a922fbf1c71cb0753e1b962c53b4d47e70"
Sep 10 16:07:07.545938 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:07.545913    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"]
Sep 10 16:07:07.551006 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:07.550985    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-247f70-openshift-default-85dd68f678-j9g6x"]
Sep 10 16:07:07.673459 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:07.673427    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45e66617-3de1-49ce-9d8b-0bc626c24765" path="/var/lib/kubelet/pods/45e66617-3de1-49ce-9d8b-0bc626c24765/volumes"
Sep 10 16:07:12.872265 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:12.872216    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-298d9b-hl6rb"]
Sep 10 16:07:12.875684 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:12.875659    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-298d9b-hl6rb"]
Sep 10 16:07:13.673221 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:13.673184    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0cdc83a-1478-4db0-9541-7d515cec437c" path="/var/lib/kubelet/pods/a0cdc83a-1478-4db0-9541-7d515cec437c/volumes"
Sep 10 16:07:20.183140 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:20.183105    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"]
Sep 10 16:07:20.183707 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:20.183341    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt" podUID="4ac0d739-9287-4fbf-a747-2ed9ae69f969" containerName="istio-proxy" containerID="cri-o://2760a249aa20e7ffd4a88d7958480de7d9e15dfeffb72df1e955d527194986b4" gracePeriod=30
Sep 10 16:07:25.443699 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.443674    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:07:25.522816 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.522744    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-data\") pod \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") "
Sep 10 16:07:25.522816 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.522783    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istiod-ca-cert\") pod \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") "
Sep 10 16:07:25.522816 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.522807    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-token\") pod \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") "
Sep 10 16:07:25.523071 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.522844    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-credential-socket\") pod \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") "
Sep 10 16:07:25.523071 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.522870    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lds47\" (UniqueName: \"kubernetes.io/projected/4ac0d739-9287-4fbf-a747-2ed9ae69f969-kube-api-access-lds47\") pod \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") "
Sep 10 16:07:25.523071 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.522920    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-workload-certs\") pod \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") "
Sep 10 16:07:25.523071 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.522971    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-envoy\") pod \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") "
Sep 10 16:07:25.523071 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.523002    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-podinfo\") pod \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") "
Sep 10 16:07:25.523071 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.523046    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-workload-socket\") pod \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\" (UID: \"4ac0d739-9287-4fbf-a747-2ed9ae69f969\") "
Sep 10 16:07:25.523381 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.523092    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-data" (OuterVolumeSpecName: "istio-data") pod "4ac0d739-9287-4fbf-a747-2ed9ae69f969" (UID: "4ac0d739-9287-4fbf-a747-2ed9ae69f969"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:25.523381 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.523234    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4ac0d739-9287-4fbf-a747-2ed9ae69f969" (UID: "4ac0d739-9287-4fbf-a747-2ed9ae69f969"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:07:25.523381 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.523257    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4ac0d739-9287-4fbf-a747-2ed9ae69f969" (UID: "4ac0d739-9287-4fbf-a747-2ed9ae69f969"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:25.523548 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.523454    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4ac0d739-9287-4fbf-a747-2ed9ae69f969" (UID: "4ac0d739-9287-4fbf-a747-2ed9ae69f969"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:25.523548 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.523469    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4ac0d739-9287-4fbf-a747-2ed9ae69f969" (UID: "4ac0d739-9287-4fbf-a747-2ed9ae69f969"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:25.523630 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.523582    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-workload-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:25.523630 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.523600    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-data\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:25.523630 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.523614    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istiod-ca-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:25.523630 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.523628    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-credential-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:25.523797 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.523639    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-workload-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:25.525259 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.525238    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-token" (OuterVolumeSpecName: "istio-token") pod "4ac0d739-9287-4fbf-a747-2ed9ae69f969" (UID: "4ac0d739-9287-4fbf-a747-2ed9ae69f969"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:07:25.525473 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.525451    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4ac0d739-9287-4fbf-a747-2ed9ae69f969" (UID: "4ac0d739-9287-4fbf-a747-2ed9ae69f969"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:07:25.525781 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.525754    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac0d739-9287-4fbf-a747-2ed9ae69f969-kube-api-access-lds47" (OuterVolumeSpecName: "kube-api-access-lds47") pod "4ac0d739-9287-4fbf-a747-2ed9ae69f969" (UID: "4ac0d739-9287-4fbf-a747-2ed9ae69f969"). InnerVolumeSpecName "kube-api-access-lds47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:07:25.525781 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.525765    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4ac0d739-9287-4fbf-a747-2ed9ae69f969" (UID: "4ac0d739-9287-4fbf-a747-2ed9ae69f969"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 16:07:25.624795 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.624758    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-envoy\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:25.624795 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.624788    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-podinfo\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:25.624795 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.624797    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac0d739-9287-4fbf-a747-2ed9ae69f969-istio-token\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:25.624795 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.624807    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lds47\" (UniqueName: \"kubernetes.io/projected/4ac0d739-9287-4fbf-a747-2ed9ae69f969-kube-api-access-lds47\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:07:25.631789 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.631760    2564 generic.go:356] "Generic (PLEG): container finished" podID="4ac0d739-9287-4fbf-a747-2ed9ae69f969" containerID="2760a249aa20e7ffd4a88d7958480de7d9e15dfeffb72df1e955d527194986b4" exitCode=0
Sep 10 16:07:25.631965 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.631818    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"
Sep 10 16:07:25.631965 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.631841    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt" event={"ID":"4ac0d739-9287-4fbf-a747-2ed9ae69f969","Type":"ContainerDied","Data":"2760a249aa20e7ffd4a88d7958480de7d9e15dfeffb72df1e955d527194986b4"}
Sep 10 16:07:25.631965 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.631897    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt" event={"ID":"4ac0d739-9287-4fbf-a747-2ed9ae69f969","Type":"ContainerDied","Data":"7858f6880bf2cb66f384c179fff3a19dc72532275f660a3e2c6f3caa0bd2285e"}
Sep 10 16:07:25.631965 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.631920    2564 scope.go:117] "RemoveContainer" containerID="2760a249aa20e7ffd4a88d7958480de7d9e15dfeffb72df1e955d527194986b4"
Sep 10 16:07:25.642296 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.642200    2564 scope.go:117] "RemoveContainer" containerID="324c47b2c289e9e8daf82a71f0bbcfdec325aa1e965e1ef459756d1647b2b3d1"
Sep 10 16:07:25.650645 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.650622    2564 scope.go:117] "RemoveContainer" containerID="2760a249aa20e7ffd4a88d7958480de7d9e15dfeffb72df1e955d527194986b4"
Sep 10 16:07:25.650892 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:07:25.650876    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2760a249aa20e7ffd4a88d7958480de7d9e15dfeffb72df1e955d527194986b4\": container with ID starting with 2760a249aa20e7ffd4a88d7958480de7d9e15dfeffb72df1e955d527194986b4 not found: ID does not exist" containerID="2760a249aa20e7ffd4a88d7958480de7d9e15dfeffb72df1e955d527194986b4"
Sep 10 16:07:25.650942 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.650901    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2760a249aa20e7ffd4a88d7958480de7d9e15dfeffb72df1e955d527194986b4"} err="failed to get container status \"2760a249aa20e7ffd4a88d7958480de7d9e15dfeffb72df1e955d527194986b4\": rpc error: code = NotFound desc = could not find container \"2760a249aa20e7ffd4a88d7958480de7d9e15dfeffb72df1e955d527194986b4\": container with ID starting with 2760a249aa20e7ffd4a88d7958480de7d9e15dfeffb72df1e955d527194986b4 not found: ID does not exist"
Sep 10 16:07:25.650942 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.650916    2564 scope.go:117] "RemoveContainer" containerID="324c47b2c289e9e8daf82a71f0bbcfdec325aa1e965e1ef459756d1647b2b3d1"
Sep 10 16:07:25.651190 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:07:25.651172    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324c47b2c289e9e8daf82a71f0bbcfdec325aa1e965e1ef459756d1647b2b3d1\": container with ID starting with 324c47b2c289e9e8daf82a71f0bbcfdec325aa1e965e1ef459756d1647b2b3d1 not found: ID does not exist" containerID="324c47b2c289e9e8daf82a71f0bbcfdec325aa1e965e1ef459756d1647b2b3d1"
Sep 10 16:07:25.651272 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.651193    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324c47b2c289e9e8daf82a71f0bbcfdec325aa1e965e1ef459756d1647b2b3d1"} err="failed to get container status \"324c47b2c289e9e8daf82a71f0bbcfdec325aa1e965e1ef459756d1647b2b3d1\": rpc error: code = NotFound desc = could not find container \"324c47b2c289e9e8daf82a71f0bbcfdec325aa1e965e1ef459756d1647b2b3d1\": container with ID starting with 324c47b2c289e9e8daf82a71f0bbcfdec325aa1e965e1ef459756d1647b2b3d1 not found: ID does not exist"
Sep 10 16:07:25.656929 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.656904    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"]
Sep 10 16:07:25.659629 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.659609    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-298d9b-openshift-default-888b56864-pjptt"]
Sep 10 16:07:25.672794 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:25.672769    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ac0d739-9287-4fbf-a747-2ed9ae69f969" path="/var/lib/kubelet/pods/4ac0d739-9287-4fbf-a747-2ed9ae69f969/volumes"
Sep 10 16:07:40.354032 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.353947    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"]
Sep 10 16:07:40.354442 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.354413    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d171cbb-6111-4067-b134-b9016a99b03f" containerName="istio-proxy"
Sep 10 16:07:40.354442 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.354425    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d171cbb-6111-4067-b134-b9016a99b03f" containerName="istio-proxy"
Sep 10 16:07:40.354442 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.354440    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e66617-3de1-49ce-9d8b-0bc626c24765" containerName="istio-proxy"
Sep 10 16:07:40.354553 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.354445    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e66617-3de1-49ce-9d8b-0bc626c24765" containerName="istio-proxy"
Sep 10 16:07:40.354553 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.354452    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e66617-3de1-49ce-9d8b-0bc626c24765" containerName="istio-proxy"
Sep 10 16:07:40.354553 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.354458    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e66617-3de1-49ce-9d8b-0bc626c24765" containerName="istio-proxy"
Sep 10 16:07:40.354553 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.354464    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ac0d739-9287-4fbf-a747-2ed9ae69f969" containerName="istio-proxy"
Sep 10 16:07:40.354553 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.354470    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ac0d739-9287-4fbf-a747-2ed9ae69f969" containerName="istio-proxy"
Sep 10 16:07:40.354553 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.354478    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0cdc83a-1478-4db0-9541-7d515cec437c" containerName="revoke-keys"
Sep 10 16:07:40.354553 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.354482    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0cdc83a-1478-4db0-9541-7d515cec437c" containerName="revoke-keys"
Sep 10 16:07:40.354553 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.354489    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ac0d739-9287-4fbf-a747-2ed9ae69f969" containerName="istio-proxy"
Sep 10 16:07:40.354553 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.354494    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ac0d739-9287-4fbf-a747-2ed9ae69f969" containerName="istio-proxy"
Sep 10 16:07:40.354825 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.354565    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ac0d739-9287-4fbf-a747-2ed9ae69f969" containerName="istio-proxy"
Sep 10 16:07:40.354825 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.354572    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d171cbb-6111-4067-b134-b9016a99b03f" containerName="istio-proxy"
Sep 10 16:07:40.354825 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.354580    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ac0d739-9287-4fbf-a747-2ed9ae69f969" containerName="istio-proxy"
Sep 10 16:07:40.354825 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.354586    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0cdc83a-1478-4db0-9541-7d515cec437c" containerName="revoke-keys"
Sep 10 16:07:40.354825 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.354593    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="45e66617-3de1-49ce-9d8b-0bc626c24765" containerName="istio-proxy"
Sep 10 16:07:40.354825 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.354598    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="45e66617-3de1-49ce-9d8b-0bc626c24765" containerName="istio-proxy"
Sep 10 16:07:40.358173 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.358141    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:40.361109 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.361081    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-108dbb\"/\"kube-root-ca.crt\""
Sep 10 16:07:40.361248 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.361083    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-108dbb\"/\"openshift-service-ca.crt\""
Sep 10 16:07:40.362326 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.362305    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-108dbb\"/\"default-dockercfg-mzb87\""
Sep 10 16:07:40.362432 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.362397    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-108dbb\"/\"rate-test-model-108dbb-kserve-self-signed-certs\""
Sep 10 16:07:40.372220 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.372194    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"]
Sep 10 16:07:40.478189 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.478124    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-dshm\") pod \"rate-test-model-108dbb-kserve-c95c69595-s8xdd\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:40.478368 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.478240    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-model-cache\") pod \"rate-test-model-108dbb-kserve-c95c69595-s8xdd\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:40.478368 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.478269    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/954b1732-01f1-41e9-8861-36faf6ff6f12-tls-certs\") pod \"rate-test-model-108dbb-kserve-c95c69595-s8xdd\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:40.478368 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.478301    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-home\") pod \"rate-test-model-108dbb-kserve-c95c69595-s8xdd\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:40.478368 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.478335    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccvpm\" (UniqueName: \"kubernetes.io/projected/954b1732-01f1-41e9-8861-36faf6ff6f12-kube-api-access-ccvpm\") pod \"rate-test-model-108dbb-kserve-c95c69595-s8xdd\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:40.478505 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.478388    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-tmp-dir\") pod \"rate-test-model-108dbb-kserve-c95c69595-s8xdd\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:40.579295 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.579261    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-dshm\") pod \"rate-test-model-108dbb-kserve-c95c69595-s8xdd\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:40.579489 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.579314    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-model-cache\") pod \"rate-test-model-108dbb-kserve-c95c69595-s8xdd\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:40.579489 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.579334    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/954b1732-01f1-41e9-8861-36faf6ff6f12-tls-certs\") pod \"rate-test-model-108dbb-kserve-c95c69595-s8xdd\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:40.579489 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.579362    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-home\") pod \"rate-test-model-108dbb-kserve-c95c69595-s8xdd\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:40.579489 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.579390    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccvpm\" (UniqueName: \"kubernetes.io/projected/954b1732-01f1-41e9-8861-36faf6ff6f12-kube-api-access-ccvpm\") pod \"rate-test-model-108dbb-kserve-c95c69595-s8xdd\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:40.579489 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.579420    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-tmp-dir\") pod \"rate-test-model-108dbb-kserve-c95c69595-s8xdd\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:40.579797 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.579774    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-home\") pod \"rate-test-model-108dbb-kserve-c95c69595-s8xdd\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:40.579854 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.579803    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-model-cache\") pod \"rate-test-model-108dbb-kserve-c95c69595-s8xdd\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:40.579854 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.579824    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-tmp-dir\") pod \"rate-test-model-108dbb-kserve-c95c69595-s8xdd\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:40.581836 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.581816    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-dshm\") pod \"rate-test-model-108dbb-kserve-c95c69595-s8xdd\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:40.581913 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.581899    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/954b1732-01f1-41e9-8861-36faf6ff6f12-tls-certs\") pod \"rate-test-model-108dbb-kserve-c95c69595-s8xdd\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:40.589748 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.589725    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccvpm\" (UniqueName: \"kubernetes.io/projected/954b1732-01f1-41e9-8861-36faf6ff6f12-kube-api-access-ccvpm\") pod \"rate-test-model-108dbb-kserve-c95c69595-s8xdd\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") " pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:40.669425 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.669399    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:40.948217 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:40.948190    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"]
Sep 10 16:07:40.949607 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:07:40.949582    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod954b1732_01f1_41e9_8861_36faf6ff6f12.slice/crio-4ef7316ee1f34417516367ab02bef961f223387ff8c2c7d0ee6fd43c13079e92 WatchSource:0}: Error finding container 4ef7316ee1f34417516367ab02bef961f223387ff8c2c7d0ee6fd43c13079e92: Status 404 returned error can't find the container with id 4ef7316ee1f34417516367ab02bef961f223387ff8c2c7d0ee6fd43c13079e92
Sep 10 16:07:41.740978 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:41.740947    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd" event={"ID":"954b1732-01f1-41e9-8861-36faf6ff6f12","Type":"ContainerStarted","Data":"ca72cea50aace0f186737b2586abba67ad7772ead1ca895733769a03601b5fa8"}
Sep 10 16:07:41.740978 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:41.740984    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd" event={"ID":"954b1732-01f1-41e9-8861-36faf6ff6f12","Type":"ContainerStarted","Data":"4ef7316ee1f34417516367ab02bef961f223387ff8c2c7d0ee6fd43c13079e92"}
Sep 10 16:07:41.763751 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:41.763703    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd" podStartSLOduration=1.763687616 podStartE2EDuration="1.763687616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:07:41.762367322 +0000 UTC m=+1540.566686586" watchObservedRunningTime="2026-09-10 16:07:41.763687616 +0000 UTC m=+1540.568006877"
Sep 10 16:07:50.670338 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:50.670306    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:50.670338 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:50.670348    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:50.682991 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:50.682969    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:07:50.806834 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:07:50.806795    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:08:27.949220 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:27.949186    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"]
Sep 10 16:08:28.712964 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.712932    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"]
Sep 10 16:08:28.718267 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.718238    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.721392 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.721372    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-fc8c1e-openshift-default-dockercfg-wfln9\""
Sep 10 16:08:28.732199 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.732173    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"]
Sep 10 16:08:28.881546 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.881510    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-data\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.881730 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.881556    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-workload-socket\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.881730 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.881584    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2348230d-7fa0-4046-9d47-29fc0fdabc70-istiod-ca-cert\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.881730 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.881608    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-credential-socket\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.881730 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.881678    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-podinfo\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.881921 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.881739    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-token\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.881921 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.881770    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-envoy\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.881921 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.881788    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzwfh\" (UniqueName: \"kubernetes.io/projected/2348230d-7fa0-4046-9d47-29fc0fdabc70-kube-api-access-mzwfh\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.881921 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.881809    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-workload-certs\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.983098 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.982992    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2348230d-7fa0-4046-9d47-29fc0fdabc70-istiod-ca-cert\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.983098 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.983046    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-credential-socket\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.983683 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.983124    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-podinfo\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.983683 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.983214    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-token\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.983683 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.983267    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-envoy\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.983683 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.983299    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzwfh\" (UniqueName: \"kubernetes.io/projected/2348230d-7fa0-4046-9d47-29fc0fdabc70-kube-api-access-mzwfh\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.983683 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.983326    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-workload-certs\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.983683 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.983383    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-data\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.983683 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.983429    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-workload-socket\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.983683 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.983524    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-credential-socket\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.984102 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.983754    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-data\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.984102 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.983751    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2348230d-7fa0-4046-9d47-29fc0fdabc70-istiod-ca-cert\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.984102 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.983824    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-workload-certs\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.984102 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.984007    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-workload-socket\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.985646 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.985616    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-envoy\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.985864 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.985846    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-podinfo\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.992778 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.992752    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-token\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:28.992944 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:28.992926    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzwfh\" (UniqueName: \"kubernetes.io/projected/2348230d-7fa0-4046-9d47-29fc0fdabc70-kube-api-access-mzwfh\") pod \"e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") " pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:29.033314 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:29.033283    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:29.316392 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:29.316360    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"]
Sep 10 16:08:29.316819 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:08:29.316784    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2348230d_7fa0_4046_9d47_29fc0fdabc70.slice/crio-6abdcc842d69cb4179a7c87ed9a4f06f304525be693032c37e5c26d6eb8a149c WatchSource:0}: Error finding container 6abdcc842d69cb4179a7c87ed9a4f06f304525be693032c37e5c26d6eb8a149c: Status 404 returned error can't find the container with id 6abdcc842d69cb4179a7c87ed9a4f06f304525be693032c37e5c26d6eb8a149c
Sep 10 16:08:29.318950 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:29.318919    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:08:29.319019 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:29.318984    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:08:29.319019 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:29.319014    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:08:30.028606 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:30.028570    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px" event={"ID":"2348230d-7fa0-4046-9d47-29fc0fdabc70","Type":"ContainerStarted","Data":"e30ec8f03d86c206c7043b1cfd3bf162405a5208bc44a09e83423194ed3c2a10"}
Sep 10 16:08:30.029052 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:30.028625    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px" event={"ID":"2348230d-7fa0-4046-9d47-29fc0fdabc70","Type":"ContainerStarted","Data":"6abdcc842d69cb4179a7c87ed9a4f06f304525be693032c37e5c26d6eb8a149c"}
Sep 10 16:08:30.033821 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:30.033802    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:30.035276 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:30.035255    2564 patch_prober.go:28] interesting pod/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 10 16:08:30.035395 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:30.035305    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px" podUID="2348230d-7fa0-4046-9d47-29fc0fdabc70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 10 16:08:30.054114 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:30.054052    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px" podStartSLOduration=2.054031334 podStartE2EDuration="2.054031334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:08:30.053233425 +0000 UTC m=+1588.857552699" watchObservedRunningTime="2026-09-10 16:08:30.054031334 +0000 UTC m=+1588.858350597"
Sep 10 16:08:31.038134 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:31.038103    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:32.040127 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:32.040097    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:32.040998 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:32.040974    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:42.970529 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:42.970468    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd" podUID="954b1732-01f1-41e9-8861-36faf6ff6f12" containerName="main" containerID="cri-o://ca72cea50aace0f186737b2586abba67ad7772ead1ca895733769a03601b5fa8" gracePeriod=45
Sep 10 16:08:43.124235 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:43.123934    2564 generic.go:356] "Generic (PLEG): container finished" podID="954b1732-01f1-41e9-8861-36faf6ff6f12" containerID="ca72cea50aace0f186737b2586abba67ad7772ead1ca895733769a03601b5fa8" exitCode=0
Sep 10 16:08:43.124235 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:43.124175    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd" event={"ID":"954b1732-01f1-41e9-8861-36faf6ff6f12","Type":"ContainerDied","Data":"ca72cea50aace0f186737b2586abba67ad7772ead1ca895733769a03601b5fa8"}
Sep 10 16:08:43.259556 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:43.259532    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:08:43.349905 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:43.349869    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccvpm\" (UniqueName: \"kubernetes.io/projected/954b1732-01f1-41e9-8861-36faf6ff6f12-kube-api-access-ccvpm\") pod \"954b1732-01f1-41e9-8861-36faf6ff6f12\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") "
Sep 10 16:08:43.349905 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:43.349912    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-tmp-dir\") pod \"954b1732-01f1-41e9-8861-36faf6ff6f12\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") "
Sep 10 16:08:43.350194 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:43.349947    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/954b1732-01f1-41e9-8861-36faf6ff6f12-tls-certs\") pod \"954b1732-01f1-41e9-8861-36faf6ff6f12\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") "
Sep 10 16:08:43.350194 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:43.350030    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-model-cache\") pod \"954b1732-01f1-41e9-8861-36faf6ff6f12\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") "
Sep 10 16:08:43.350194 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:43.350052    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-home\") pod \"954b1732-01f1-41e9-8861-36faf6ff6f12\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") "
Sep 10 16:08:43.350194 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:43.350117    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-dshm\") pod \"954b1732-01f1-41e9-8861-36faf6ff6f12\" (UID: \"954b1732-01f1-41e9-8861-36faf6ff6f12\") "
Sep 10 16:08:43.350194 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:43.350187    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "954b1732-01f1-41e9-8861-36faf6ff6f12" (UID: "954b1732-01f1-41e9-8861-36faf6ff6f12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:08:43.350479 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:43.350259    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-model-cache" (OuterVolumeSpecName: "model-cache") pod "954b1732-01f1-41e9-8861-36faf6ff6f12" (UID: "954b1732-01f1-41e9-8861-36faf6ff6f12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:08:43.350479 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:43.350300    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-home" (OuterVolumeSpecName: "home") pod "954b1732-01f1-41e9-8861-36faf6ff6f12" (UID: "954b1732-01f1-41e9-8861-36faf6ff6f12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:08:43.350582 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:43.350477    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-tmp-dir\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:08:43.350582 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:43.350494    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-model-cache\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:08:43.350582 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:43.350502    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-home\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:08:43.352308 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:43.352279    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954b1732-01f1-41e9-8861-36faf6ff6f12-kube-api-access-ccvpm" (OuterVolumeSpecName: "kube-api-access-ccvpm") pod "954b1732-01f1-41e9-8861-36faf6ff6f12" (UID: "954b1732-01f1-41e9-8861-36faf6ff6f12"). InnerVolumeSpecName "kube-api-access-ccvpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:08:43.352412 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:43.352288    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-dshm" (OuterVolumeSpecName: "dshm") pod "954b1732-01f1-41e9-8861-36faf6ff6f12" (UID: "954b1732-01f1-41e9-8861-36faf6ff6f12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:08:43.352451 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:43.352418    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/954b1732-01f1-41e9-8861-36faf6ff6f12-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "954b1732-01f1-41e9-8861-36faf6ff6f12" (UID: "954b1732-01f1-41e9-8861-36faf6ff6f12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:08:43.451270 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:43.451237    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/954b1732-01f1-41e9-8861-36faf6ff6f12-dshm\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:08:43.451270 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:43.451268    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccvpm\" (UniqueName: \"kubernetes.io/projected/954b1732-01f1-41e9-8861-36faf6ff6f12-kube-api-access-ccvpm\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:08:43.451270 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:43.451279    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/954b1732-01f1-41e9-8861-36faf6ff6f12-tls-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:08:44.131941 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:44.131842    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd" event={"ID":"954b1732-01f1-41e9-8861-36faf6ff6f12","Type":"ContainerDied","Data":"4ef7316ee1f34417516367ab02bef961f223387ff8c2c7d0ee6fd43c13079e92"}
Sep 10 16:08:44.131941 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:44.131880    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"
Sep 10 16:08:44.131941 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:44.131898    2564 scope.go:117] "RemoveContainer" containerID="ca72cea50aace0f186737b2586abba67ad7772ead1ca895733769a03601b5fa8"
Sep 10 16:08:44.152898 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:44.152871    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"]
Sep 10 16:08:44.159239 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:44.159196    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-108dbb/rate-test-model-108dbb-kserve-c95c69595-s8xdd"]
Sep 10 16:08:45.672952 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:45.672919    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="954b1732-01f1-41e9-8861-36faf6ff6f12" path="/var/lib/kubelet/pods/954b1732-01f1-41e9-8861-36faf6ff6f12/volumes"
Sep 10 16:08:47.897789 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:47.897729    2564 patch_prober.go:28] interesting pod/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": read tcp 10.133.0.2:56874->10.133.0.90:15021: read: connection reset by peer" start-of-body=
Sep 10 16:08:47.898289 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:47.897814    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px" podUID="2348230d-7fa0-4046-9d47-29fc0fdabc70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": read tcp 10.133.0.2:56874->10.133.0.90:15021: read: connection reset by peer"
Sep 10 16:08:48.158309 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:48.158216    2564 generic.go:356] "Generic (PLEG): container finished" podID="2348230d-7fa0-4046-9d47-29fc0fdabc70" containerID="e30ec8f03d86c206c7043b1cfd3bf162405a5208bc44a09e83423194ed3c2a10" exitCode=0
Sep 10 16:08:48.158309 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:48.158295    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px" event={"ID":"2348230d-7fa0-4046-9d47-29fc0fdabc70","Type":"ContainerDied","Data":"e30ec8f03d86c206c7043b1cfd3bf162405a5208bc44a09e83423194ed3c2a10"}
Sep 10 16:08:48.158770 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:48.158755    2564 scope.go:117] "RemoveContainer" containerID="e30ec8f03d86c206c7043b1cfd3bf162405a5208bc44a09e83423194ed3c2a10"
Sep 10 16:08:48.159921 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:48.159883    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:08:48.159979 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:48.159967    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:08:48.160020 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:48.160001    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:08:49.165996 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:49.165968    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px" event={"ID":"2348230d-7fa0-4046-9d47-29fc0fdabc70","Type":"ContainerStarted","Data":"fb049c17dd7b6d7f653ab0c8aad7b95bed99e80238277836d1baae46f097502d"}
Sep 10 16:08:50.033720 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:50.033685    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:50.033913 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:50.033794    2564 patch_prober.go:28] interesting pod/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 10 16:08:50.033913 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:50.033828    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px" podUID="2348230d-7fa0-4046-9d47-29fc0fdabc70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 10 16:08:51.034138 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:51.034102    2564 patch_prober.go:28] interesting pod/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 10 16:08:51.034549 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:51.034182    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px" podUID="2348230d-7fa0-4046-9d47-29fc0fdabc70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 10 16:08:52.034017 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:52.033971    2564 patch_prober.go:28] interesting pod/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 10 16:08:52.034242 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:52.034050    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px" podUID="2348230d-7fa0-4046-9d47-29fc0fdabc70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 10 16:08:53.034506 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:53.034422    2564 patch_prober.go:28] interesting pod/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 10 16:08:53.034506 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:53.034496    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px" podUID="2348230d-7fa0-4046-9d47-29fc0fdabc70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 10 16:08:54.033857 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:54.033817    2564 patch_prober.go:28] interesting pod/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 10 16:08:54.034053 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:54.033878    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px" podUID="2348230d-7fa0-4046-9d47-29fc0fdabc70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 10 16:08:55.037469 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:55.037437    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:55.037915 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:55.037832    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:55.038687 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:55.038668    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:08:57.920574 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:57.920539    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"]
Sep 10 16:08:57.921259 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:57.921241    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="954b1732-01f1-41e9-8861-36faf6ff6f12" containerName="main"
Sep 10 16:08:57.921325 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:57.921261    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="954b1732-01f1-41e9-8861-36faf6ff6f12" containerName="main"
Sep 10 16:08:57.921362 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:57.921354    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="954b1732-01f1-41e9-8861-36faf6ff6f12" containerName="main"
Sep 10 16:08:57.923661 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:57.923642    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:57.926310 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:57.926293    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ddc985-openshift-default-dockercfg-9ksrd\""
Sep 10 16:08:57.943374 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:57.943348    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"]
Sep 10 16:08:58.105182 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.105131    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-istio-data\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.105407 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.105206    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-workload-certs\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.105407 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.105228    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-istio-envoy\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.105407 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.105255    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a167e343-148c-4d12-93fe-49287be965f2-istio-token\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.105407 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.105310    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-workload-socket\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.105407 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.105328    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-credential-socket\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.105407 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.105390    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a167e343-148c-4d12-93fe-49287be965f2-istiod-ca-cert\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.105652 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.105410    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tmzk\" (UniqueName: \"kubernetes.io/projected/a167e343-148c-4d12-93fe-49287be965f2-kube-api-access-4tmzk\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.105652 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.105432    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a167e343-148c-4d12-93fe-49287be965f2-istio-podinfo\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.206383 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.206294    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-workload-socket\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.206627 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.206601    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-credential-socket\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.206798 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.206781    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a167e343-148c-4d12-93fe-49287be965f2-istiod-ca-cert\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.206927 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.206910    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tmzk\" (UniqueName: \"kubernetes.io/projected/a167e343-148c-4d12-93fe-49287be965f2-kube-api-access-4tmzk\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.207067 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.207050    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a167e343-148c-4d12-93fe-49287be965f2-istio-podinfo\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.207224 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.207205    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-credential-socket\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.207308 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.206931    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-workload-socket\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.208109 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.208080    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-istio-data\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.208270 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.208136    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-workload-certs\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.208270 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.208044    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a167e343-148c-4d12-93fe-49287be965f2-istiod-ca-cert\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.208407 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.208277    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-istio-envoy\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.208407 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.208337    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a167e343-148c-4d12-93fe-49287be965f2-istio-token\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.208522 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.208440    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-istio-data\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.208707 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.208681    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-workload-certs\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.210514 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.210491    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a167e343-148c-4d12-93fe-49287be965f2-istio-podinfo\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.211034 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.211017    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-istio-envoy\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.218181 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.218139    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a167e343-148c-4d12-93fe-49287be965f2-istio-token\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.218650 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.218631    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tmzk\" (UniqueName: \"kubernetes.io/projected/a167e343-148c-4d12-93fe-49287be965f2-kube-api-access-4tmzk\") pod \"e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") " pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.233607 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.233586    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:58.522513 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.522489    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"]
Sep 10 16:08:58.524213 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:08:58.524188    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda167e343_148c_4d12_93fe_49287be965f2.slice/crio-7a1688828eaa099aa9ac843a89eb008845b4aa0c1a83501bdd984278e17e7900 WatchSource:0}: Error finding container 7a1688828eaa099aa9ac843a89eb008845b4aa0c1a83501bdd984278e17e7900: Status 404 returned error can't find the container with id 7a1688828eaa099aa9ac843a89eb008845b4aa0c1a83501bdd984278e17e7900
Sep 10 16:08:58.526498 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.526464    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:08:58.526569 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.526535    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:08:58.526650 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:58.526568    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:08:59.227732 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:59.227701    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq" event={"ID":"a167e343-148c-4d12-93fe-49287be965f2","Type":"ContainerStarted","Data":"06ab476235c623dce16c6fc6c72d9d34cd3d533d26de7379fa33d025fcf07a19"}
Sep 10 16:08:59.228100 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:59.227751    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq" event={"ID":"a167e343-148c-4d12-93fe-49287be965f2","Type":"ContainerStarted","Data":"7a1688828eaa099aa9ac843a89eb008845b4aa0c1a83501bdd984278e17e7900"}
Sep 10 16:08:59.234453 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:59.234429    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:59.239874 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:59.239855    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:08:59.254816 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:08:59.254773    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq" podStartSLOduration=2.254759421 podStartE2EDuration="2.254759421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:08:59.253845836 +0000 UTC m=+1618.058165105" watchObservedRunningTime="2026-09-10 16:08:59.254759421 +0000 UTC m=+1618.059078682"
Sep 10 16:09:00.233325 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:00.233292    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:09:00.234432 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:00.234413    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:09:02.461382 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:02.461346    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"]
Sep 10 16:09:02.464343 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:02.464310    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"
Sep 10 16:09:02.467991 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:02.467963    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qfdsq\""
Sep 10 16:09:02.468148 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:02.467964    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ddc985-serving-cert\""
Sep 10 16:09:02.468148 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:02.467965    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 16:09:02.482746 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:02.482710    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"]
Sep 10 16:09:02.550023 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:02.549979    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-metrics-certs\") pod \"maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm\" (UID: \"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"
Sep 10 16:09:02.550268 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:02.550050    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8t89\" (UniqueName: \"kubernetes.io/projected/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-kube-api-access-p8t89\") pod \"maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm\" (UID: \"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"
Sep 10 16:09:02.550268 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:02.550094    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm\" (UID: \"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"
Sep 10 16:09:02.550268 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:02.550120    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm\" (UID: \"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"
Sep 10 16:09:02.651747 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:02.651706    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-metrics-certs\") pod \"maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm\" (UID: \"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"
Sep 10 16:09:02.651968 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:02.651772    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8t89\" (UniqueName: \"kubernetes.io/projected/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-kube-api-access-p8t89\") pod \"maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm\" (UID: \"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"
Sep 10 16:09:02.651968 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:02.651795    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm\" (UID: \"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"
Sep 10 16:09:02.651968 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:02.651816    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm\" (UID: \"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"
Sep 10 16:09:02.652531 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:02.652512    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm\" (UID: \"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"
Sep 10 16:09:02.654448 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:02.654425    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-metrics-certs\") pod \"maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm\" (UID: \"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"
Sep 10 16:09:02.654448 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:02.654445    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm\" (UID: \"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"
Sep 10 16:09:02.661369 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:02.661342    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8t89\" (UniqueName: \"kubernetes.io/projected/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-kube-api-access-p8t89\") pod \"maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm\" (UID: \"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"
Sep 10 16:09:02.709193 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:02.709126    2564 scope.go:117] "RemoveContainer" containerID="c559940d367b28e9cadbd42c0bad1da97df5ddf7d3c13f5e6bbcc7fcc4b9ee2c"
Sep 10 16:09:02.776537 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:02.776505    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"
Sep 10 16:09:03.062822 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:03.062793    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"]
Sep 10 16:09:03.065104 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:09:03.065062    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5d6092_b2a0_4910_80c5_6227d4c0f2d3.slice/crio-5fe96f54242b45e03ee0d1c2a1f6e670617cb05ad592c1d0df073f060e986311 WatchSource:0}: Error finding container 5fe96f54242b45e03ee0d1c2a1f6e670617cb05ad592c1d0df073f060e986311: Status 404 returned error can't find the container with id 5fe96f54242b45e03ee0d1c2a1f6e670617cb05ad592c1d0df073f060e986311
Sep 10 16:09:03.066694 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:03.066679    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 16:09:03.252710 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:03.252673    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm" event={"ID":"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3","Type":"ContainerStarted","Data":"5fe96f54242b45e03ee0d1c2a1f6e670617cb05ad592c1d0df073f060e986311"}
Sep 10 16:09:05.265851 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:05.265814    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm" event={"ID":"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3","Type":"ContainerStarted","Data":"6e5b5cfe1c53406c3bd36c7ce31d8c2fe9eac2c12a3bdd65a2bd2ee617defad3"}
Sep 10 16:09:05.266279 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:05.265995    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"
Sep 10 16:09:05.287004 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:05.286955    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm" podStartSLOduration=1.791642538 podStartE2EDuration="3.286940334s" totalImagesPullingTime="1.495297796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:09:05.285351413 +0000 UTC m=+1624.089670675" watchObservedRunningTime="2026-09-10 16:09:05.286940334 +0000 UTC m=+1624.091259595"
Sep 10 16:09:11.278463 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:11.278433    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"
Sep 10 16:09:20.369221 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:20.369147    2564 generic.go:356] "Generic (PLEG): container finished" podID="a167e343-148c-4d12-93fe-49287be965f2" containerID="06ab476235c623dce16c6fc6c72d9d34cd3d533d26de7379fa33d025fcf07a19" exitCode=0
Sep 10 16:09:20.369655 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:20.369259    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq" event={"ID":"a167e343-148c-4d12-93fe-49287be965f2","Type":"ContainerDied","Data":"06ab476235c623dce16c6fc6c72d9d34cd3d533d26de7379fa33d025fcf07a19"}
Sep 10 16:09:20.369707 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:20.369694    2564 scope.go:117] "RemoveContainer" containerID="06ab476235c623dce16c6fc6c72d9d34cd3d533d26de7379fa33d025fcf07a19"
Sep 10 16:09:20.370948 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:20.370919    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:09:20.371028 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:20.370991    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:09:20.371028 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:20.371023    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 16:09:21.377329 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:21.377295    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq" event={"ID":"a167e343-148c-4d12-93fe-49287be965f2","Type":"ContainerStarted","Data":"a78327460fdb64917f10b299533634159a155b39c7029cb633ea592bf1621d73"}
Sep 10 16:09:22.234453 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:22.234417    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:09:22.234715 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:22.234691    2564 patch_prober.go:28] interesting pod/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Sep 10 16:09:22.234772 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:22.234748    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq" podUID="a167e343-148c-4d12-93fe-49287be965f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Sep 10 16:09:23.234501 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:23.234463    2564 patch_prober.go:28] interesting pod/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Sep 10 16:09:23.234920 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:23.234525    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq" podUID="a167e343-148c-4d12-93fe-49287be965f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Sep 10 16:09:24.235024 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:24.234982    2564 patch_prober.go:28] interesting pod/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Sep 10 16:09:24.235566 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:24.235085    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq" podUID="a167e343-148c-4d12-93fe-49287be965f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Sep 10 16:09:25.234064 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:25.234019    2564 patch_prober.go:28] interesting pod/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Sep 10 16:09:25.234321 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:25.234090    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq" podUID="a167e343-148c-4d12-93fe-49287be965f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Sep 10 16:09:26.234409 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:26.234376    2564 patch_prober.go:28] interesting pod/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Sep 10 16:09:26.234811 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:26.234432    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq" podUID="a167e343-148c-4d12-93fe-49287be965f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Sep 10 16:09:27.238340 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:27.238304    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:09:27.238810 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:27.238716    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:09:27.239446 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:27.239424    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:09:31.173858 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:09:31.173824    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4n4q2_d551ac05-c043-4e13-90da-c8c978d709f7/payload-processing/0.log"
Sep 10 16:10:22.358607 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:22.358574    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4n4q2_d551ac05-c043-4e13-90da-c8c978d709f7/payload-processing/0.log"
Sep 10 16:10:27.145514 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:27.145480    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"]
Sep 10 16:10:27.149769 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:27.149743    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"
Sep 10 16:10:27.152243 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:27.152215    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-65ee1d-serving-cert\""
Sep 10 16:10:27.160496 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:27.160469    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"]
Sep 10 16:10:27.249928 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:27.249882    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50af11b5-0b01-4279-ba35-d1f5e87a496f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f\" (UID: \"50af11b5-0b01-4279-ba35-d1f5e87a496f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"
Sep 10 16:10:27.250128 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:27.249936    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50af11b5-0b01-4279-ba35-d1f5e87a496f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f\" (UID: \"50af11b5-0b01-4279-ba35-d1f5e87a496f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"
Sep 10 16:10:27.250128 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:27.250035    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50af11b5-0b01-4279-ba35-d1f5e87a496f-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f\" (UID: \"50af11b5-0b01-4279-ba35-d1f5e87a496f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"
Sep 10 16:10:27.250128 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:27.250121    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlvgf\" (UniqueName: \"kubernetes.io/projected/50af11b5-0b01-4279-ba35-d1f5e87a496f-kube-api-access-wlvgf\") pod \"maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f\" (UID: \"50af11b5-0b01-4279-ba35-d1f5e87a496f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"
Sep 10 16:10:27.350870 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:27.350844    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50af11b5-0b01-4279-ba35-d1f5e87a496f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f\" (UID: \"50af11b5-0b01-4279-ba35-d1f5e87a496f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"
Sep 10 16:10:27.351064 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:27.350877    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50af11b5-0b01-4279-ba35-d1f5e87a496f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f\" (UID: \"50af11b5-0b01-4279-ba35-d1f5e87a496f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"
Sep 10 16:10:27.351064 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:27.350906    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50af11b5-0b01-4279-ba35-d1f5e87a496f-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f\" (UID: \"50af11b5-0b01-4279-ba35-d1f5e87a496f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"
Sep 10 16:10:27.351064 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:27.350946    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlvgf\" (UniqueName: \"kubernetes.io/projected/50af11b5-0b01-4279-ba35-d1f5e87a496f-kube-api-access-wlvgf\") pod \"maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f\" (UID: \"50af11b5-0b01-4279-ba35-d1f5e87a496f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"
Sep 10 16:10:27.351549 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:27.351527    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50af11b5-0b01-4279-ba35-d1f5e87a496f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f\" (UID: \"50af11b5-0b01-4279-ba35-d1f5e87a496f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"
Sep 10 16:10:27.353610 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:27.353590    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50af11b5-0b01-4279-ba35-d1f5e87a496f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f\" (UID: \"50af11b5-0b01-4279-ba35-d1f5e87a496f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"
Sep 10 16:10:27.353610 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:27.353602    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50af11b5-0b01-4279-ba35-d1f5e87a496f-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f\" (UID: \"50af11b5-0b01-4279-ba35-d1f5e87a496f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"
Sep 10 16:10:27.361260 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:27.361237    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlvgf\" (UniqueName: \"kubernetes.io/projected/50af11b5-0b01-4279-ba35-d1f5e87a496f-kube-api-access-wlvgf\") pod \"maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f\" (UID: \"50af11b5-0b01-4279-ba35-d1f5e87a496f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"
Sep 10 16:10:27.464294 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:27.464261    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"
Sep 10 16:10:27.744349 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:27.744314    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"]
Sep 10 16:10:27.746228 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:10:27.746148    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50af11b5_0b01_4279_ba35_d1f5e87a496f.slice/crio-251185bbcc962157a2cdc4b6935e7e10862ec341eeace76dce28990f2dbcc24b WatchSource:0}: Error finding container 251185bbcc962157a2cdc4b6935e7e10862ec341eeace76dce28990f2dbcc24b: Status 404 returned error can't find the container with id 251185bbcc962157a2cdc4b6935e7e10862ec341eeace76dce28990f2dbcc24b
Sep 10 16:10:27.770735 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:27.770694    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f" event={"ID":"50af11b5-0b01-4279-ba35-d1f5e87a496f","Type":"ContainerStarted","Data":"251185bbcc962157a2cdc4b6935e7e10862ec341eeace76dce28990f2dbcc24b"}
Sep 10 16:10:28.779961 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:28.779924    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f" event={"ID":"50af11b5-0b01-4279-ba35-d1f5e87a496f","Type":"ContainerStarted","Data":"0229b82cbc7a6975a6651008451d72d6429334f0a588879695df17133d26f6b5"}
Sep 10 16:10:28.780485 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:28.779996    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"
Sep 10 16:10:28.810454 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:28.810397    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f" podStartSLOduration=1.256315466 podStartE2EDuration="1.810378511s" totalImagesPullingTime="554.063045ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:10:28.808537525 +0000 UTC m=+1707.612856786" watchObservedRunningTime="2026-09-10 16:10:28.810378511 +0000 UTC m=+1707.614697772"
Sep 10 16:10:34.791362 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:34.791330    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"
Sep 10 16:10:57.778558 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:57.778523    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"]
Sep 10 16:10:57.779447 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:57.779407    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f" podUID="50af11b5-0b01-4279-ba35-d1f5e87a496f" containerName="maas-api" containerID="cri-o://0229b82cbc7a6975a6651008451d72d6429334f0a588879695df17133d26f6b5" gracePeriod=30
Sep 10 16:10:57.953511 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:57.953200    2564 generic.go:356] "Generic (PLEG): container finished" podID="50af11b5-0b01-4279-ba35-d1f5e87a496f" containerID="0229b82cbc7a6975a6651008451d72d6429334f0a588879695df17133d26f6b5" exitCode=0
Sep 10 16:10:57.953511 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:57.953369    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f" event={"ID":"50af11b5-0b01-4279-ba35-d1f5e87a496f","Type":"ContainerDied","Data":"0229b82cbc7a6975a6651008451d72d6429334f0a588879695df17133d26f6b5"}
Sep 10 16:10:58.044683 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:58.044659    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"
Sep 10 16:10:58.169479 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:58.169448    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50af11b5-0b01-4279-ba35-d1f5e87a496f-openshift-service-ca\") pod \"50af11b5-0b01-4279-ba35-d1f5e87a496f\" (UID: \"50af11b5-0b01-4279-ba35-d1f5e87a496f\") "
Sep 10 16:10:58.169674 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:58.169568    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50af11b5-0b01-4279-ba35-d1f5e87a496f-maas-api-tls\") pod \"50af11b5-0b01-4279-ba35-d1f5e87a496f\" (UID: \"50af11b5-0b01-4279-ba35-d1f5e87a496f\") "
Sep 10 16:10:58.169674 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:58.169615    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50af11b5-0b01-4279-ba35-d1f5e87a496f-metrics-certs\") pod \"50af11b5-0b01-4279-ba35-d1f5e87a496f\" (UID: \"50af11b5-0b01-4279-ba35-d1f5e87a496f\") "
Sep 10 16:10:58.169674 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:58.169646    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlvgf\" (UniqueName: \"kubernetes.io/projected/50af11b5-0b01-4279-ba35-d1f5e87a496f-kube-api-access-wlvgf\") pod \"50af11b5-0b01-4279-ba35-d1f5e87a496f\" (UID: \"50af11b5-0b01-4279-ba35-d1f5e87a496f\") "
Sep 10 16:10:58.169841 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:58.169817    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50af11b5-0b01-4279-ba35-d1f5e87a496f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "50af11b5-0b01-4279-ba35-d1f5e87a496f" (UID: "50af11b5-0b01-4279-ba35-d1f5e87a496f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:10:58.170071 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:58.170048    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50af11b5-0b01-4279-ba35-d1f5e87a496f-openshift-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:10:58.172027 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:58.172000    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50af11b5-0b01-4279-ba35-d1f5e87a496f-kube-api-access-wlvgf" (OuterVolumeSpecName: "kube-api-access-wlvgf") pod "50af11b5-0b01-4279-ba35-d1f5e87a496f" (UID: "50af11b5-0b01-4279-ba35-d1f5e87a496f"). InnerVolumeSpecName "kube-api-access-wlvgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:10:58.172360 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:58.172339    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50af11b5-0b01-4279-ba35-d1f5e87a496f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "50af11b5-0b01-4279-ba35-d1f5e87a496f" (UID: "50af11b5-0b01-4279-ba35-d1f5e87a496f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:10:58.172454 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:58.172389    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50af11b5-0b01-4279-ba35-d1f5e87a496f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "50af11b5-0b01-4279-ba35-d1f5e87a496f" (UID: "50af11b5-0b01-4279-ba35-d1f5e87a496f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:10:58.271429 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:58.271386    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50af11b5-0b01-4279-ba35-d1f5e87a496f-maas-api-tls\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:10:58.271429 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:58.271424    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50af11b5-0b01-4279-ba35-d1f5e87a496f-metrics-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:10:58.271429 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:58.271437    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlvgf\" (UniqueName: \"kubernetes.io/projected/50af11b5-0b01-4279-ba35-d1f5e87a496f-kube-api-access-wlvgf\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:10:58.967844 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:58.967808    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f" event={"ID":"50af11b5-0b01-4279-ba35-d1f5e87a496f","Type":"ContainerDied","Data":"251185bbcc962157a2cdc4b6935e7e10862ec341eeace76dce28990f2dbcc24b"}
Sep 10 16:10:58.968316 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:58.967825    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"
Sep 10 16:10:58.968316 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:58.967910    2564 scope.go:117] "RemoveContainer" containerID="0229b82cbc7a6975a6651008451d72d6429334f0a588879695df17133d26f6b5"
Sep 10 16:10:58.991268 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:58.991237    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"]
Sep 10 16:10:58.995888 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:58.995867    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-65ee1d-85d77fc989-g5p7f"]
Sep 10 16:10:59.673361 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:10:59.673331    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50af11b5-0b01-4279-ba35-d1f5e87a496f" path="/var/lib/kubelet/pods/50af11b5-0b01-4279-ba35-d1f5e87a496f/volumes"
Sep 10 16:12:02.488762 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:02.488736    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-snm4z_f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f/console-operator/1.log"
Sep 10 16:12:02.491954 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:02.491932    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-snm4z_f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f/console-operator/1.log"
Sep 10 16:12:02.758382 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:02.758289    2564 scope.go:117] "RemoveContainer" containerID="8dd96e721e2b11fbdcbde09ee3ef9b3cbf9bd1047cf5a592494a974fdf692b54"
Sep 10 16:12:14.856103 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:14.856070    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"]
Sep 10 16:12:14.856548 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:14.856330    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px" podUID="2348230d-7fa0-4046-9d47-29fc0fdabc70" containerName="istio-proxy" containerID="cri-o://fb049c17dd7b6d7f653ab0c8aad7b95bed99e80238277836d1baae46f097502d" gracePeriod=30
Sep 10 16:12:20.110929 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.110898    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:12:20.135496 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.135403    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-credential-socket\") pod \"2348230d-7fa0-4046-9d47-29fc0fdabc70\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") "
Sep 10 16:12:20.135496 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.135466    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-podinfo\") pod \"2348230d-7fa0-4046-9d47-29fc0fdabc70\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") "
Sep 10 16:12:20.135726 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.135510    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2348230d-7fa0-4046-9d47-29fc0fdabc70-istiod-ca-cert\") pod \"2348230d-7fa0-4046-9d47-29fc0fdabc70\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") "
Sep 10 16:12:20.135726 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.135558    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-workload-socket\") pod \"2348230d-7fa0-4046-9d47-29fc0fdabc70\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") "
Sep 10 16:12:20.135726 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.135579    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-envoy\") pod \"2348230d-7fa0-4046-9d47-29fc0fdabc70\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") "
Sep 10 16:12:20.135927 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.135858    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2348230d-7fa0-4046-9d47-29fc0fdabc70" (UID: "2348230d-7fa0-4046-9d47-29fc0fdabc70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:12:20.136596 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.136053    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzwfh\" (UniqueName: \"kubernetes.io/projected/2348230d-7fa0-4046-9d47-29fc0fdabc70-kube-api-access-mzwfh\") pod \"2348230d-7fa0-4046-9d47-29fc0fdabc70\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") "
Sep 10 16:12:20.136596 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.136078    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2348230d-7fa0-4046-9d47-29fc0fdabc70" (UID: "2348230d-7fa0-4046-9d47-29fc0fdabc70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:12:20.136596 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.136109    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2348230d-7fa0-4046-9d47-29fc0fdabc70-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2348230d-7fa0-4046-9d47-29fc0fdabc70" (UID: "2348230d-7fa0-4046-9d47-29fc0fdabc70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:12:20.136596 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.136180    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-data\") pod \"2348230d-7fa0-4046-9d47-29fc0fdabc70\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") "
Sep 10 16:12:20.136596 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.136220    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-workload-certs\") pod \"2348230d-7fa0-4046-9d47-29fc0fdabc70\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") "
Sep 10 16:12:20.136596 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.136254    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-token\") pod \"2348230d-7fa0-4046-9d47-29fc0fdabc70\" (UID: \"2348230d-7fa0-4046-9d47-29fc0fdabc70\") "
Sep 10 16:12:20.136596 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.136564    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2348230d-7fa0-4046-9d47-29fc0fdabc70" (UID: "2348230d-7fa0-4046-9d47-29fc0fdabc70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:12:20.137026 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.136619    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-data" (OuterVolumeSpecName: "istio-data") pod "2348230d-7fa0-4046-9d47-29fc0fdabc70" (UID: "2348230d-7fa0-4046-9d47-29fc0fdabc70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:12:20.137026 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.136680    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-workload-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:20.137026 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.136699    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-credential-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:20.137026 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.136715    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2348230d-7fa0-4046-9d47-29fc0fdabc70-istiod-ca-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:20.137026 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.136728    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-workload-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:20.139001 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.138976    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2348230d-7fa0-4046-9d47-29fc0fdabc70" (UID: "2348230d-7fa0-4046-9d47-29fc0fdabc70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 16:12:20.139138 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.139026    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2348230d-7fa0-4046-9d47-29fc0fdabc70" (UID: "2348230d-7fa0-4046-9d47-29fc0fdabc70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:12:20.139138 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.139114    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2348230d-7fa0-4046-9d47-29fc0fdabc70-kube-api-access-mzwfh" (OuterVolumeSpecName: "kube-api-access-mzwfh") pod "2348230d-7fa0-4046-9d47-29fc0fdabc70" (UID: "2348230d-7fa0-4046-9d47-29fc0fdabc70"). InnerVolumeSpecName "kube-api-access-mzwfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:12:20.139362 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.139335    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-token" (OuterVolumeSpecName: "istio-token") pod "2348230d-7fa0-4046-9d47-29fc0fdabc70" (UID: "2348230d-7fa0-4046-9d47-29fc0fdabc70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:12:20.237897 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.237864    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-envoy\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:20.237897 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.237893    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzwfh\" (UniqueName: \"kubernetes.io/projected/2348230d-7fa0-4046-9d47-29fc0fdabc70-kube-api-access-mzwfh\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:20.237897 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.237902    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-data\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:20.238133 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.237910    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-token\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:20.238133 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.237919    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2348230d-7fa0-4046-9d47-29fc0fdabc70-istio-podinfo\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:20.444389 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.444352    2564 generic.go:356] "Generic (PLEG): container finished" podID="2348230d-7fa0-4046-9d47-29fc0fdabc70" containerID="fb049c17dd7b6d7f653ab0c8aad7b95bed99e80238277836d1baae46f097502d" exitCode=0
Sep 10 16:12:20.444569 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.444407    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"
Sep 10 16:12:20.444569 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.444440    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px" event={"ID":"2348230d-7fa0-4046-9d47-29fc0fdabc70","Type":"ContainerDied","Data":"fb049c17dd7b6d7f653ab0c8aad7b95bed99e80238277836d1baae46f097502d"}
Sep 10 16:12:20.444569 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.444531    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px" event={"ID":"2348230d-7fa0-4046-9d47-29fc0fdabc70","Type":"ContainerDied","Data":"6abdcc842d69cb4179a7c87ed9a4f06f304525be693032c37e5c26d6eb8a149c"}
Sep 10 16:12:20.444569 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.444556    2564 scope.go:117] "RemoveContainer" containerID="fb049c17dd7b6d7f653ab0c8aad7b95bed99e80238277836d1baae46f097502d"
Sep 10 16:12:20.454683 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.454662    2564 scope.go:117] "RemoveContainer" containerID="e30ec8f03d86c206c7043b1cfd3bf162405a5208bc44a09e83423194ed3c2a10"
Sep 10 16:12:20.463415 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.463394    2564 scope.go:117] "RemoveContainer" containerID="fb049c17dd7b6d7f653ab0c8aad7b95bed99e80238277836d1baae46f097502d"
Sep 10 16:12:20.463666 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:12:20.463640    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb049c17dd7b6d7f653ab0c8aad7b95bed99e80238277836d1baae46f097502d\": container with ID starting with fb049c17dd7b6d7f653ab0c8aad7b95bed99e80238277836d1baae46f097502d not found: ID does not exist" containerID="fb049c17dd7b6d7f653ab0c8aad7b95bed99e80238277836d1baae46f097502d"
Sep 10 16:12:20.463745 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.463679    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb049c17dd7b6d7f653ab0c8aad7b95bed99e80238277836d1baae46f097502d"} err="failed to get container status \"fb049c17dd7b6d7f653ab0c8aad7b95bed99e80238277836d1baae46f097502d\": rpc error: code = NotFound desc = could not find container \"fb049c17dd7b6d7f653ab0c8aad7b95bed99e80238277836d1baae46f097502d\": container with ID starting with fb049c17dd7b6d7f653ab0c8aad7b95bed99e80238277836d1baae46f097502d not found: ID does not exist"
Sep 10 16:12:20.463745 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.463698    2564 scope.go:117] "RemoveContainer" containerID="e30ec8f03d86c206c7043b1cfd3bf162405a5208bc44a09e83423194ed3c2a10"
Sep 10 16:12:20.463932 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:12:20.463917    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e30ec8f03d86c206c7043b1cfd3bf162405a5208bc44a09e83423194ed3c2a10\": container with ID starting with e30ec8f03d86c206c7043b1cfd3bf162405a5208bc44a09e83423194ed3c2a10 not found: ID does not exist" containerID="e30ec8f03d86c206c7043b1cfd3bf162405a5208bc44a09e83423194ed3c2a10"
Sep 10 16:12:20.463986 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.463938    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30ec8f03d86c206c7043b1cfd3bf162405a5208bc44a09e83423194ed3c2a10"} err="failed to get container status \"e30ec8f03d86c206c7043b1cfd3bf162405a5208bc44a09e83423194ed3c2a10\": rpc error: code = NotFound desc = could not find container \"e30ec8f03d86c206c7043b1cfd3bf162405a5208bc44a09e83423194ed3c2a10\": container with ID starting with e30ec8f03d86c206c7043b1cfd3bf162405a5208bc44a09e83423194ed3c2a10 not found: ID does not exist"
Sep 10 16:12:20.469211 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.469188    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"]
Sep 10 16:12:20.477274 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:20.477244    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-fc8c1e-openshift-default-54f87c7774-zn7px"]
Sep 10 16:12:21.673300 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:21.673271    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2348230d-7fa0-4046-9d47-29fc0fdabc70" path="/var/lib/kubelet/pods/2348230d-7fa0-4046-9d47-29fc0fdabc70/volumes"
Sep 10 16:12:25.838207 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:25.838173    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"]
Sep 10 16:12:25.838684 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:25.838380    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm" podUID="ad5d6092-b2a0-4910-80c5-6227d4c0f2d3" containerName="maas-api" containerID="cri-o://6e5b5cfe1c53406c3bd36c7ce31d8c2fe9eac2c12a3bdd65a2bd2ee617defad3" gracePeriod=30
Sep 10 16:12:26.103179 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.103093    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"
Sep 10 16:12:26.199970 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.199935    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-metrics-certs\") pod \"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3\" (UID: \"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3\") "
Sep 10 16:12:26.200200 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.199980    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-openshift-service-ca\") pod \"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3\" (UID: \"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3\") "
Sep 10 16:12:26.200200 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.200108    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-maas-api-tls\") pod \"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3\" (UID: \"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3\") "
Sep 10 16:12:26.200200 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.200181    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8t89\" (UniqueName: \"kubernetes.io/projected/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-kube-api-access-p8t89\") pod \"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3\" (UID: \"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3\") "
Sep 10 16:12:26.200462 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.200433    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ad5d6092-b2a0-4910-80c5-6227d4c0f2d3" (UID: "ad5d6092-b2a0-4910-80c5-6227d4c0f2d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:12:26.200546 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.200528    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-openshift-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:26.202534 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.202498    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ad5d6092-b2a0-4910-80c5-6227d4c0f2d3" (UID: "ad5d6092-b2a0-4910-80c5-6227d4c0f2d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:12:26.202986 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.202965    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-kube-api-access-p8t89" (OuterVolumeSpecName: "kube-api-access-p8t89") pod "ad5d6092-b2a0-4910-80c5-6227d4c0f2d3" (UID: "ad5d6092-b2a0-4910-80c5-6227d4c0f2d3"). InnerVolumeSpecName "kube-api-access-p8t89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:12:26.202986 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.202967    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ad5d6092-b2a0-4910-80c5-6227d4c0f2d3" (UID: "ad5d6092-b2a0-4910-80c5-6227d4c0f2d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:12:26.302088 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.302055    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8t89\" (UniqueName: \"kubernetes.io/projected/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-kube-api-access-p8t89\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:26.302088 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.302084    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-metrics-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:26.302088 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.302093    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3-maas-api-tls\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:26.480246 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.480214    2564 generic.go:356] "Generic (PLEG): container finished" podID="ad5d6092-b2a0-4910-80c5-6227d4c0f2d3" containerID="6e5b5cfe1c53406c3bd36c7ce31d8c2fe9eac2c12a3bdd65a2bd2ee617defad3" exitCode=0
Sep 10 16:12:26.480453 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.480258    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"
Sep 10 16:12:26.480453 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.480298    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm" event={"ID":"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3","Type":"ContainerDied","Data":"6e5b5cfe1c53406c3bd36c7ce31d8c2fe9eac2c12a3bdd65a2bd2ee617defad3"}
Sep 10 16:12:26.480453 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.480389    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm" event={"ID":"ad5d6092-b2a0-4910-80c5-6227d4c0f2d3","Type":"ContainerDied","Data":"5fe96f54242b45e03ee0d1c2a1f6e670617cb05ad592c1d0df073f060e986311"}
Sep 10 16:12:26.480453 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.480419    2564 scope.go:117] "RemoveContainer" containerID="6e5b5cfe1c53406c3bd36c7ce31d8c2fe9eac2c12a3bdd65a2bd2ee617defad3"
Sep 10 16:12:26.490537 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.490521    2564 scope.go:117] "RemoveContainer" containerID="6e5b5cfe1c53406c3bd36c7ce31d8c2fe9eac2c12a3bdd65a2bd2ee617defad3"
Sep 10 16:12:26.490832 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:12:26.490810    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e5b5cfe1c53406c3bd36c7ce31d8c2fe9eac2c12a3bdd65a2bd2ee617defad3\": container with ID starting with 6e5b5cfe1c53406c3bd36c7ce31d8c2fe9eac2c12a3bdd65a2bd2ee617defad3 not found: ID does not exist" containerID="6e5b5cfe1c53406c3bd36c7ce31d8c2fe9eac2c12a3bdd65a2bd2ee617defad3"
Sep 10 16:12:26.490882 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.490842    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e5b5cfe1c53406c3bd36c7ce31d8c2fe9eac2c12a3bdd65a2bd2ee617defad3"} err="failed to get container status \"6e5b5cfe1c53406c3bd36c7ce31d8c2fe9eac2c12a3bdd65a2bd2ee617defad3\": rpc error: code = NotFound desc = could not find container \"6e5b5cfe1c53406c3bd36c7ce31d8c2fe9eac2c12a3bdd65a2bd2ee617defad3\": container with ID starting with 6e5b5cfe1c53406c3bd36c7ce31d8c2fe9eac2c12a3bdd65a2bd2ee617defad3 not found: ID does not exist"
Sep 10 16:12:26.505266 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.505242    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"]
Sep 10 16:12:26.512452 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:26.512429    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc985-76f9b58844-5f8sm"]
Sep 10 16:12:27.673475 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:27.673441    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad5d6092-b2a0-4910-80c5-6227d4c0f2d3" path="/var/lib/kubelet/pods/ad5d6092-b2a0-4910-80c5-6227d4c0f2d3/volumes"
Sep 10 16:12:33.194792 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:33.194757    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"]
Sep 10 16:12:33.195375 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:33.195042    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq" podUID="a167e343-148c-4d12-93fe-49287be965f2" containerName="istio-proxy" containerID="cri-o://a78327460fdb64917f10b299533634159a155b39c7029cb633ea592bf1621d73" gracePeriod=30
Sep 10 16:12:38.457367 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.457305    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:12:38.549876 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.549842    2564 generic.go:356] "Generic (PLEG): container finished" podID="a167e343-148c-4d12-93fe-49287be965f2" containerID="a78327460fdb64917f10b299533634159a155b39c7029cb633ea592bf1621d73" exitCode=0
Sep 10 16:12:38.550049 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.549887    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"
Sep 10 16:12:38.550049 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.549917    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq" event={"ID":"a167e343-148c-4d12-93fe-49287be965f2","Type":"ContainerDied","Data":"a78327460fdb64917f10b299533634159a155b39c7029cb633ea592bf1621d73"}
Sep 10 16:12:38.550049 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.549964    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq" event={"ID":"a167e343-148c-4d12-93fe-49287be965f2","Type":"ContainerDied","Data":"7a1688828eaa099aa9ac843a89eb008845b4aa0c1a83501bdd984278e17e7900"}
Sep 10 16:12:38.550049 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.549986    2564 scope.go:117] "RemoveContainer" containerID="a78327460fdb64917f10b299533634159a155b39c7029cb633ea592bf1621d73"
Sep 10 16:12:38.559452 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.559432    2564 scope.go:117] "RemoveContainer" containerID="06ab476235c623dce16c6fc6c72d9d34cd3d533d26de7379fa33d025fcf07a19"
Sep 10 16:12:38.567831 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.567815    2564 scope.go:117] "RemoveContainer" containerID="a78327460fdb64917f10b299533634159a155b39c7029cb633ea592bf1621d73"
Sep 10 16:12:38.568099 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:12:38.568079    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a78327460fdb64917f10b299533634159a155b39c7029cb633ea592bf1621d73\": container with ID starting with a78327460fdb64917f10b299533634159a155b39c7029cb633ea592bf1621d73 not found: ID does not exist" containerID="a78327460fdb64917f10b299533634159a155b39c7029cb633ea592bf1621d73"
Sep 10 16:12:38.568236 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.568106    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a78327460fdb64917f10b299533634159a155b39c7029cb633ea592bf1621d73"} err="failed to get container status \"a78327460fdb64917f10b299533634159a155b39c7029cb633ea592bf1621d73\": rpc error: code = NotFound desc = could not find container \"a78327460fdb64917f10b299533634159a155b39c7029cb633ea592bf1621d73\": container with ID starting with a78327460fdb64917f10b299533634159a155b39c7029cb633ea592bf1621d73 not found: ID does not exist"
Sep 10 16:12:38.568236 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.568123    2564 scope.go:117] "RemoveContainer" containerID="06ab476235c623dce16c6fc6c72d9d34cd3d533d26de7379fa33d025fcf07a19"
Sep 10 16:12:38.568434 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:12:38.568418    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ab476235c623dce16c6fc6c72d9d34cd3d533d26de7379fa33d025fcf07a19\": container with ID starting with 06ab476235c623dce16c6fc6c72d9d34cd3d533d26de7379fa33d025fcf07a19 not found: ID does not exist" containerID="06ab476235c623dce16c6fc6c72d9d34cd3d533d26de7379fa33d025fcf07a19"
Sep 10 16:12:38.568483 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.568439    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ab476235c623dce16c6fc6c72d9d34cd3d533d26de7379fa33d025fcf07a19"} err="failed to get container status \"06ab476235c623dce16c6fc6c72d9d34cd3d533d26de7379fa33d025fcf07a19\": rpc error: code = NotFound desc = could not find container \"06ab476235c623dce16c6fc6c72d9d34cd3d533d26de7379fa33d025fcf07a19\": container with ID starting with 06ab476235c623dce16c6fc6c72d9d34cd3d533d26de7379fa33d025fcf07a19 not found: ID does not exist"
Sep 10 16:12:38.627189 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.627129    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a167e343-148c-4d12-93fe-49287be965f2-istio-podinfo\") pod \"a167e343-148c-4d12-93fe-49287be965f2\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") "
Sep 10 16:12:38.627374 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.627219    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-workload-certs\") pod \"a167e343-148c-4d12-93fe-49287be965f2\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") "
Sep 10 16:12:38.627374 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.627268    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-credential-socket\") pod \"a167e343-148c-4d12-93fe-49287be965f2\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") "
Sep 10 16:12:38.627374 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.627285    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-workload-socket\") pod \"a167e343-148c-4d12-93fe-49287be965f2\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") "
Sep 10 16:12:38.627374 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.627352    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a167e343-148c-4d12-93fe-49287be965f2-istio-token\") pod \"a167e343-148c-4d12-93fe-49287be965f2\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") "
Sep 10 16:12:38.627374 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.627374    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-istio-envoy\") pod \"a167e343-148c-4d12-93fe-49287be965f2\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") "
Sep 10 16:12:38.627638 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.627410    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a167e343-148c-4d12-93fe-49287be965f2-istiod-ca-cert\") pod \"a167e343-148c-4d12-93fe-49287be965f2\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") "
Sep 10 16:12:38.627638 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.627451    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-istio-data\") pod \"a167e343-148c-4d12-93fe-49287be965f2\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") "
Sep 10 16:12:38.627638 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.627475    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tmzk\" (UniqueName: \"kubernetes.io/projected/a167e343-148c-4d12-93fe-49287be965f2-kube-api-access-4tmzk\") pod \"a167e343-148c-4d12-93fe-49287be965f2\" (UID: \"a167e343-148c-4d12-93fe-49287be965f2\") "
Sep 10 16:12:38.627638 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.627568    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a167e343-148c-4d12-93fe-49287be965f2" (UID: "a167e343-148c-4d12-93fe-49287be965f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:12:38.627638 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.627579    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a167e343-148c-4d12-93fe-49287be965f2" (UID: "a167e343-148c-4d12-93fe-49287be965f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:12:38.627638 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.627610    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a167e343-148c-4d12-93fe-49287be965f2" (UID: "a167e343-148c-4d12-93fe-49287be965f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:12:38.627939 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.627770    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-istio-data" (OuterVolumeSpecName: "istio-data") pod "a167e343-148c-4d12-93fe-49287be965f2" (UID: "a167e343-148c-4d12-93fe-49287be965f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:12:38.627939 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.627926    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-istio-data\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:38.628232 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.627945    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-workload-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:38.628232 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.627959    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-credential-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:38.628232 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.627973    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-workload-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:38.628232 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.628053    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a167e343-148c-4d12-93fe-49287be965f2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a167e343-148c-4d12-93fe-49287be965f2" (UID: "a167e343-148c-4d12-93fe-49287be965f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:12:38.629599 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.629569    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a167e343-148c-4d12-93fe-49287be965f2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a167e343-148c-4d12-93fe-49287be965f2" (UID: "a167e343-148c-4d12-93fe-49287be965f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 16:12:38.629768 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.629750    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a167e343-148c-4d12-93fe-49287be965f2-kube-api-access-4tmzk" (OuterVolumeSpecName: "kube-api-access-4tmzk") pod "a167e343-148c-4d12-93fe-49287be965f2" (UID: "a167e343-148c-4d12-93fe-49287be965f2"). InnerVolumeSpecName "kube-api-access-4tmzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:12:38.629836 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.629763    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a167e343-148c-4d12-93fe-49287be965f2-istio-token" (OuterVolumeSpecName: "istio-token") pod "a167e343-148c-4d12-93fe-49287be965f2" (UID: "a167e343-148c-4d12-93fe-49287be965f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:12:38.630179 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.630136    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a167e343-148c-4d12-93fe-49287be965f2" (UID: "a167e343-148c-4d12-93fe-49287be965f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:12:38.729033 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.728938    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a167e343-148c-4d12-93fe-49287be965f2-istio-podinfo\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:38.729033 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.728968    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a167e343-148c-4d12-93fe-49287be965f2-istio-token\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:38.729033 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.728981    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a167e343-148c-4d12-93fe-49287be965f2-istio-envoy\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:38.729033 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.728991    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a167e343-148c-4d12-93fe-49287be965f2-istiod-ca-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:38.729033 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.729003    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tmzk\" (UniqueName: \"kubernetes.io/projected/a167e343-148c-4d12-93fe-49287be965f2-kube-api-access-4tmzk\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:38.874553 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.874524    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"]
Sep 10 16:12:38.880347 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:38.880320    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ddc985-openshift-default-5f6dc89c78-hx6jq"]
Sep 10 16:12:39.673999 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:39.673960    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a167e343-148c-4d12-93fe-49287be965f2" path="/var/lib/kubelet/pods/a167e343-148c-4d12-93fe-49287be965f2/volumes"
Sep 10 16:12:53.527196 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.527147    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"]
Sep 10 16:12:53.527630 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.527389    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6" podUID="6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" containerName="istio-proxy" containerID="cri-o://409769f954d4370ace4b83feb561472d0dd2003afdac323e15ade8525bc13dd3" gracePeriod=30
Sep 10 16:12:53.922336 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.922298    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-108dbb-xpl57"]
Sep 10 16:12:53.922736 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.922723    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a167e343-148c-4d12-93fe-49287be965f2" containerName="istio-proxy"
Sep 10 16:12:53.922780 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.922737    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a167e343-148c-4d12-93fe-49287be965f2" containerName="istio-proxy"
Sep 10 16:12:53.922780 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.922752    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2348230d-7fa0-4046-9d47-29fc0fdabc70" containerName="istio-proxy"
Sep 10 16:12:53.922780 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.922757    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2348230d-7fa0-4046-9d47-29fc0fdabc70" containerName="istio-proxy"
Sep 10 16:12:53.922780 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.922764    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2348230d-7fa0-4046-9d47-29fc0fdabc70" containerName="istio-proxy"
Sep 10 16:12:53.922780 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.922770    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2348230d-7fa0-4046-9d47-29fc0fdabc70" containerName="istio-proxy"
Sep 10 16:12:53.922780 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.922780    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50af11b5-0b01-4279-ba35-d1f5e87a496f" containerName="maas-api"
Sep 10 16:12:53.922967 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.922785    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="50af11b5-0b01-4279-ba35-d1f5e87a496f" containerName="maas-api"
Sep 10 16:12:53.922967 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.922801    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad5d6092-b2a0-4910-80c5-6227d4c0f2d3" containerName="maas-api"
Sep 10 16:12:53.922967 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.922806    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad5d6092-b2a0-4910-80c5-6227d4c0f2d3" containerName="maas-api"
Sep 10 16:12:53.922967 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.922864    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2348230d-7fa0-4046-9d47-29fc0fdabc70" containerName="istio-proxy"
Sep 10 16:12:53.922967 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.922871    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="50af11b5-0b01-4279-ba35-d1f5e87a496f" containerName="maas-api"
Sep 10 16:12:53.922967 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.922878    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a167e343-148c-4d12-93fe-49287be965f2" containerName="istio-proxy"
Sep 10 16:12:53.922967 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.922883    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a167e343-148c-4d12-93fe-49287be965f2" containerName="istio-proxy"
Sep 10 16:12:53.922967 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.922889    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad5d6092-b2a0-4910-80c5-6227d4c0f2d3" containerName="maas-api"
Sep 10 16:12:53.935996 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.935969    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-108dbb-xpl57"
Sep 10 16:12:53.938929 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.938900    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2wghg\""
Sep 10 16:12:53.940029 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.940005    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-108dbb-xpl57"]
Sep 10 16:12:53.990844 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:53.990817    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c51b354-74f5-4893-8489-1ef7ab1f7a9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-108dbb-xpl57\" (UID: \"9c51b354-74f5-4893-8489-1ef7ab1f7a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-108dbb-xpl57"
Sep 10 16:12:54.092203 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:54.092150    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c51b354-74f5-4893-8489-1ef7ab1f7a9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-108dbb-xpl57\" (UID: \"9c51b354-74f5-4893-8489-1ef7ab1f7a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-108dbb-xpl57"
Sep 10 16:12:54.092841 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:54.092820    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c51b354-74f5-4893-8489-1ef7ab1f7a9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-108dbb-xpl57\" (UID: \"9c51b354-74f5-4893-8489-1ef7ab1f7a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-108dbb-xpl57"
Sep 10 16:12:54.249850 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:54.249760    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-108dbb-xpl57"
Sep 10 16:12:54.537368 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:54.537281    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-108dbb-xpl57"]
Sep 10 16:12:54.540666 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:12:54.540638    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c51b354_74f5_4893_8489_1ef7ab1f7a9a.slice/crio-c95bc384d48a740402c48b4206de95c4c6cadd6d2a96bc13152514a5c4761d43 WatchSource:0}: Error finding container c95bc384d48a740402c48b4206de95c4c6cadd6d2a96bc13152514a5c4761d43: Status 404 returned error can't find the container with id c95bc384d48a740402c48b4206de95c4c6cadd6d2a96bc13152514a5c4761d43
Sep 10 16:12:54.647855 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:54.647811    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-108dbb-xpl57" event={"ID":"9c51b354-74f5-4893-8489-1ef7ab1f7a9a","Type":"ContainerStarted","Data":"c95bc384d48a740402c48b4206de95c4c6cadd6d2a96bc13152514a5c4761d43"}
Sep 10 16:12:55.655751 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:55.655656    2564 generic.go:356] "Generic (PLEG): container finished" podID="9c51b354-74f5-4893-8489-1ef7ab1f7a9a" containerID="200cd1df8d0114efd78b4105b5c156716ebe4147cc6cecc6e1eb184fd689deda" exitCode=0
Sep 10 16:12:55.655751 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:55.655719    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-108dbb-xpl57" event={"ID":"9c51b354-74f5-4893-8489-1ef7ab1f7a9a","Type":"ContainerDied","Data":"200cd1df8d0114efd78b4105b5c156716ebe4147cc6cecc6e1eb184fd689deda"}
Sep 10 16:12:56.801173 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:56.801133    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-108dbb-xpl57"
Sep 10 16:12:56.817607 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:56.817582    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c51b354-74f5-4893-8489-1ef7ab1f7a9a-maas-api-service-ca\") pod \"9c51b354-74f5-4893-8489-1ef7ab1f7a9a\" (UID: \"9c51b354-74f5-4893-8489-1ef7ab1f7a9a\") "
Sep 10 16:12:56.817898 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:56.817877    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c51b354-74f5-4893-8489-1ef7ab1f7a9a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9c51b354-74f5-4893-8489-1ef7ab1f7a9a" (UID: "9c51b354-74f5-4893-8489-1ef7ab1f7a9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:12:56.919064 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:56.919032    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c51b354-74f5-4893-8489-1ef7ab1f7a9a-maas-api-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:57.668283 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:57.668253    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-108dbb-xpl57"
Sep 10 16:12:57.674004 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:57.673973    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-108dbb-xpl57" event={"ID":"9c51b354-74f5-4893-8489-1ef7ab1f7a9a","Type":"ContainerDied","Data":"c95bc384d48a740402c48b4206de95c4c6cadd6d2a96bc13152514a5c4761d43"}
Sep 10 16:12:57.674142 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:57.674020    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c95bc384d48a740402c48b4206de95c4c6cadd6d2a96bc13152514a5c4761d43"
Sep 10 16:12:58.676976 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.676945    2564 generic.go:356] "Generic (PLEG): container finished" podID="6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" containerID="409769f954d4370ace4b83feb561472d0dd2003afdac323e15ade8525bc13dd3" exitCode=0
Sep 10 16:12:58.677387 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.677018    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6" event={"ID":"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8","Type":"ContainerDied","Data":"409769f954d4370ace4b83feb561472d0dd2003afdac323e15ade8525bc13dd3"}
Sep 10 16:12:58.677387 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.677080    2564 scope.go:117] "RemoveContainer" containerID="c0db2a278ab1ec8b3ece0e2b3f407f4902107354689d110a9f33477f8b907a0b"
Sep 10 16:12:58.800349 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.800289    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:12:58.837867 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.837609    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-workload-certs\") pod \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") "
Sep 10 16:12:58.837867 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.837686    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-credential-socket\") pod \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") "
Sep 10 16:12:58.837867 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.837710    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-envoy\") pod \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") "
Sep 10 16:12:58.837867 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.837738    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-data\") pod \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") "
Sep 10 16:12:58.837867 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.837788    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntrvz\" (UniqueName: \"kubernetes.io/projected/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-kube-api-access-ntrvz\") pod \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") "
Sep 10 16:12:58.838333 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.837881    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istiod-ca-cert\") pod \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") "
Sep 10 16:12:58.838333 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.837921    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-workload-socket\") pod \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") "
Sep 10 16:12:58.838333 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.837949    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-podinfo\") pod \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") "
Sep 10 16:12:58.838333 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.837944    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" (UID: "6fe09b0b-462a-4b7a-a95b-8ed91494a7e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:12:58.838333 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.837982    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-token\") pod \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\" (UID: \"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8\") "
Sep 10 16:12:58.838333 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.838081    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-data" (OuterVolumeSpecName: "istio-data") pod "6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" (UID: "6fe09b0b-462a-4b7a-a95b-8ed91494a7e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:12:58.838627 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.838356    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-workload-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:58.838627 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.838376    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-data\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:58.838627 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.838539    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" (UID: "6fe09b0b-462a-4b7a-a95b-8ed91494a7e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:12:58.838952 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.838691    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" (UID: "6fe09b0b-462a-4b7a-a95b-8ed91494a7e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:12:58.838952 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.838912    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" (UID: "6fe09b0b-462a-4b7a-a95b-8ed91494a7e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:12:58.840792 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.840728    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-token" (OuterVolumeSpecName: "istio-token") pod "6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" (UID: "6fe09b0b-462a-4b7a-a95b-8ed91494a7e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:12:58.840934 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.840855    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-kube-api-access-ntrvz" (OuterVolumeSpecName: "kube-api-access-ntrvz") pod "6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" (UID: "6fe09b0b-462a-4b7a-a95b-8ed91494a7e8"). InnerVolumeSpecName "kube-api-access-ntrvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:12:58.841007 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.840985    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" (UID: "6fe09b0b-462a-4b7a-a95b-8ed91494a7e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 16:12:58.841140 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.841121    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" (UID: "6fe09b0b-462a-4b7a-a95b-8ed91494a7e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:12:58.939544 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.939506    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istiod-ca-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:58.939544 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.939537    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-workload-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:58.939544 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.939547    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-podinfo\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:58.939544 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.939555    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-token\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:58.939818 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.939586    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-credential-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:58.939818 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.939595    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-istio-envoy\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:58.939818 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:58.939603    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntrvz\" (UniqueName: \"kubernetes.io/projected/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8-kube-api-access-ntrvz\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:12:59.684006 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:59.683978    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6" event={"ID":"6fe09b0b-462a-4b7a-a95b-8ed91494a7e8","Type":"ContainerDied","Data":"077fa8f85e1d9fa75b003b0fa394fcb2c132dc5192c3e308caf32b16e9d37a6a"}
Sep 10 16:12:59.684006 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:59.684000    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"
Sep 10 16:12:59.684482 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:59.684031    2564 scope.go:117] "RemoveContainer" containerID="409769f954d4370ace4b83feb561472d0dd2003afdac323e15ade8525bc13dd3"
Sep 10 16:12:59.710853 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:59.710826    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"]
Sep 10 16:12:59.717271 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:12:59.717248    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b52270-openshift-default-77f5c8988-hc6l6"]
Sep 10 16:13:01.673781 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:01.673744    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" path="/var/lib/kubelet/pods/6fe09b0b-462a-4b7a-a95b-8ed91494a7e8/volumes"
Sep 10 16:13:02.780679 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:02.780651    2564 scope.go:117] "RemoveContainer" containerID="92784fecd04efce2bd51da36aacd32428393fef66c42ca7314eba77fa29688ba"
Sep 10 16:13:02.789229 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:02.789210    2564 scope.go:117] "RemoveContainer" containerID="5756186284b14d8d7cabd0e519d6388f971fb2c1e2cbbde3019debcdf9c56fd9"
Sep 10 16:13:02.796944 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:02.796924    2564 scope.go:117] "RemoveContainer" containerID="6198e53616a8f5d77157927855756c3243e57bb66c9f92e1dceb016f9332a229"
Sep 10 16:13:05.068964 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:05.068915    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-108dbb-xpl57"]
Sep 10 16:13:05.071599 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:05.071565    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-108dbb-xpl57"]
Sep 10 16:13:05.674049 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:05.674004    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c51b354-74f5-4893-8489-1ef7ab1f7a9a" path="/var/lib/kubelet/pods/9c51b354-74f5-4893-8489-1ef7ab1f7a9a/volumes"
Sep 10 16:13:13.030629 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:13.030591    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"]
Sep 10 16:13:13.031127 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:13.030875    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw" podUID="f413d5cd-e143-4982-8e07-ede979564650" containerName="istio-proxy" containerID="cri-o://67cf9c48a3294a238e76f09eaec91c3259ace60af98b802cdf524e6ddd6de644" gracePeriod=30
Sep 10 16:13:15.674086 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:15.674053    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cd9db79db-rndpc"]
Sep 10 16:13:15.674505 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:15.674312    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5cd9db79db-rndpc" podUID="373a49de-5c83-48d4-a4d9-43d1b5ed6558" containerName="manager" containerID="cri-o://d2d1b3b51aa8e4eaab44103a39052ca0487e36ddd31975baa7835cba2f9acc25" gracePeriod=30
Sep 10 16:13:15.937288 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:15.937223    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd9db79db-rndpc"
Sep 10 16:13:16.021807 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:16.021774    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373a49de-5c83-48d4-a4d9-43d1b5ed6558-metrics-certs\") pod \"373a49de-5c83-48d4-a4d9-43d1b5ed6558\" (UID: \"373a49de-5c83-48d4-a4d9-43d1b5ed6558\") "
Sep 10 16:13:16.022015 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:16.021875    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnhtr\" (UniqueName: \"kubernetes.io/projected/373a49de-5c83-48d4-a4d9-43d1b5ed6558-kube-api-access-bnhtr\") pod \"373a49de-5c83-48d4-a4d9-43d1b5ed6558\" (UID: \"373a49de-5c83-48d4-a4d9-43d1b5ed6558\") "
Sep 10 16:13:16.022015 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:16.021954    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/373a49de-5c83-48d4-a4d9-43d1b5ed6558-webhook-cert\") pod \"373a49de-5c83-48d4-a4d9-43d1b5ed6558\" (UID: \"373a49de-5c83-48d4-a4d9-43d1b5ed6558\") "
Sep 10 16:13:16.024177 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:16.024120    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373a49de-5c83-48d4-a4d9-43d1b5ed6558-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "373a49de-5c83-48d4-a4d9-43d1b5ed6558" (UID: "373a49de-5c83-48d4-a4d9-43d1b5ed6558"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:13:16.024177 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:16.024147    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373a49de-5c83-48d4-a4d9-43d1b5ed6558-kube-api-access-bnhtr" (OuterVolumeSpecName: "kube-api-access-bnhtr") pod "373a49de-5c83-48d4-a4d9-43d1b5ed6558" (UID: "373a49de-5c83-48d4-a4d9-43d1b5ed6558"). InnerVolumeSpecName "kube-api-access-bnhtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:13:16.024328 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:16.024149    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373a49de-5c83-48d4-a4d9-43d1b5ed6558-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "373a49de-5c83-48d4-a4d9-43d1b5ed6558" (UID: "373a49de-5c83-48d4-a4d9-43d1b5ed6558"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:13:16.123902 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:16.123860    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnhtr\" (UniqueName: \"kubernetes.io/projected/373a49de-5c83-48d4-a4d9-43d1b5ed6558-kube-api-access-bnhtr\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:13:16.123902 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:16.123895    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/373a49de-5c83-48d4-a4d9-43d1b5ed6558-webhook-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:13:16.123902 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:16.123905    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373a49de-5c83-48d4-a4d9-43d1b5ed6558-metrics-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:13:16.788795 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:16.788755    2564 generic.go:356] "Generic (PLEG): container finished" podID="373a49de-5c83-48d4-a4d9-43d1b5ed6558" containerID="d2d1b3b51aa8e4eaab44103a39052ca0487e36ddd31975baa7835cba2f9acc25" exitCode=0
Sep 10 16:13:16.789307 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:16.788820    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd9db79db-rndpc"
Sep 10 16:13:16.789307 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:16.788840    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd9db79db-rndpc" event={"ID":"373a49de-5c83-48d4-a4d9-43d1b5ed6558","Type":"ContainerDied","Data":"d2d1b3b51aa8e4eaab44103a39052ca0487e36ddd31975baa7835cba2f9acc25"}
Sep 10 16:13:16.789307 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:16.788898    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd9db79db-rndpc" event={"ID":"373a49de-5c83-48d4-a4d9-43d1b5ed6558","Type":"ContainerDied","Data":"fbc6988ef8e414515c12ef3fcf15ba9ed1b6af85370b5e8cdd369ea9e1100065"}
Sep 10 16:13:16.789307 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:16.788922    2564 scope.go:117] "RemoveContainer" containerID="d2d1b3b51aa8e4eaab44103a39052ca0487e36ddd31975baa7835cba2f9acc25"
Sep 10 16:13:16.799728 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:16.799699    2564 scope.go:117] "RemoveContainer" containerID="d2d1b3b51aa8e4eaab44103a39052ca0487e36ddd31975baa7835cba2f9acc25"
Sep 10 16:13:16.800103 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:13:16.800075    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2d1b3b51aa8e4eaab44103a39052ca0487e36ddd31975baa7835cba2f9acc25\": container with ID starting with d2d1b3b51aa8e4eaab44103a39052ca0487e36ddd31975baa7835cba2f9acc25 not found: ID does not exist" containerID="d2d1b3b51aa8e4eaab44103a39052ca0487e36ddd31975baa7835cba2f9acc25"
Sep 10 16:13:16.800217 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:16.800113    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2d1b3b51aa8e4eaab44103a39052ca0487e36ddd31975baa7835cba2f9acc25"} err="failed to get container status \"d2d1b3b51aa8e4eaab44103a39052ca0487e36ddd31975baa7835cba2f9acc25\": rpc error: code = NotFound desc = could not find container \"d2d1b3b51aa8e4eaab44103a39052ca0487e36ddd31975baa7835cba2f9acc25\": container with ID starting with d2d1b3b51aa8e4eaab44103a39052ca0487e36ddd31975baa7835cba2f9acc25 not found: ID does not exist"
Sep 10 16:13:16.814285 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:16.814242    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cd9db79db-rndpc"]
Sep 10 16:13:16.820630 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:16.820603    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5cd9db79db-rndpc"]
Sep 10 16:13:17.672986 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:17.672943    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="373a49de-5c83-48d4-a4d9-43d1b5ed6558" path="/var/lib/kubelet/pods/373a49de-5c83-48d4-a4d9-43d1b5ed6558/volumes"
Sep 10 16:13:18.316397 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.316307    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:13:18.448838 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.448801    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-credential-socket\") pod \"f413d5cd-e143-4982-8e07-ede979564650\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") "
Sep 10 16:13:18.449041 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.448868    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-workload-socket\") pod \"f413d5cd-e143-4982-8e07-ede979564650\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") "
Sep 10 16:13:18.449041 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.448899    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f413d5cd-e143-4982-8e07-ede979564650-istiod-ca-cert\") pod \"f413d5cd-e143-4982-8e07-ede979564650\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") "
Sep 10 16:13:18.449041 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.448940    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hndgn\" (UniqueName: \"kubernetes.io/projected/f413d5cd-e143-4982-8e07-ede979564650-kube-api-access-hndgn\") pod \"f413d5cd-e143-4982-8e07-ede979564650\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") "
Sep 10 16:13:18.449041 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.448979    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-istio-envoy\") pod \"f413d5cd-e143-4982-8e07-ede979564650\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") "
Sep 10 16:13:18.449041 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.449014    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-istio-data\") pod \"f413d5cd-e143-4982-8e07-ede979564650\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") "
Sep 10 16:13:18.449348 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.449055    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f413d5cd-e143-4982-8e07-ede979564650-istio-podinfo\") pod \"f413d5cd-e143-4982-8e07-ede979564650\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") "
Sep 10 16:13:18.449348 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.449090    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f413d5cd-e143-4982-8e07-ede979564650" (UID: "f413d5cd-e143-4982-8e07-ede979564650"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:13:18.449348 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.449112    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-workload-certs\") pod \"f413d5cd-e143-4982-8e07-ede979564650\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") "
Sep 10 16:13:18.449348 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.449196    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f413d5cd-e143-4982-8e07-ede979564650-istio-token\") pod \"f413d5cd-e143-4982-8e07-ede979564650\" (UID: \"f413d5cd-e143-4982-8e07-ede979564650\") "
Sep 10 16:13:18.449348 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.449323    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-istio-data" (OuterVolumeSpecName: "istio-data") pod "f413d5cd-e143-4982-8e07-ede979564650" (UID: "f413d5cd-e143-4982-8e07-ede979564650"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:13:18.449612 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.449337    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f413d5cd-e143-4982-8e07-ede979564650-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f413d5cd-e143-4982-8e07-ede979564650" (UID: "f413d5cd-e143-4982-8e07-ede979564650"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:13:18.449612 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.449384    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f413d5cd-e143-4982-8e07-ede979564650" (UID: "f413d5cd-e143-4982-8e07-ede979564650"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:13:18.449612 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.449595    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-credential-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:13:18.449743 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.449614    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-workload-socket\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:13:18.449743 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.449630    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f413d5cd-e143-4982-8e07-ede979564650" (UID: "f413d5cd-e143-4982-8e07-ede979564650"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:13:18.449743 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.449639    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f413d5cd-e143-4982-8e07-ede979564650-istiod-ca-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:13:18.449743 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.449653    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-istio-data\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:13:18.452057 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.452032    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f413d5cd-e143-4982-8e07-ede979564650-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f413d5cd-e143-4982-8e07-ede979564650" (UID: "f413d5cd-e143-4982-8e07-ede979564650"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 16:13:18.452057 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.452036    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f413d5cd-e143-4982-8e07-ede979564650" (UID: "f413d5cd-e143-4982-8e07-ede979564650"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:13:18.452253 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.452058    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f413d5cd-e143-4982-8e07-ede979564650-kube-api-access-hndgn" (OuterVolumeSpecName: "kube-api-access-hndgn") pod "f413d5cd-e143-4982-8e07-ede979564650" (UID: "f413d5cd-e143-4982-8e07-ede979564650"). InnerVolumeSpecName "kube-api-access-hndgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:13:18.452253 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.452038    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f413d5cd-e143-4982-8e07-ede979564650-istio-token" (OuterVolumeSpecName: "istio-token") pod "f413d5cd-e143-4982-8e07-ede979564650" (UID: "f413d5cd-e143-4982-8e07-ede979564650"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:13:18.550274 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.550236    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-workload-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:13:18.550274 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.550266    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f413d5cd-e143-4982-8e07-ede979564650-istio-token\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:13:18.550274 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.550275    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hndgn\" (UniqueName: \"kubernetes.io/projected/f413d5cd-e143-4982-8e07-ede979564650-kube-api-access-hndgn\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:13:18.550274 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.550284    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f413d5cd-e143-4982-8e07-ede979564650-istio-envoy\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:13:18.550551 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.550292    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f413d5cd-e143-4982-8e07-ede979564650-istio-podinfo\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:13:18.803764 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.803719    2564 generic.go:356] "Generic (PLEG): container finished" podID="f413d5cd-e143-4982-8e07-ede979564650" containerID="67cf9c48a3294a238e76f09eaec91c3259ace60af98b802cdf524e6ddd6de644" exitCode=0
Sep 10 16:13:18.803964 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.803805    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw" event={"ID":"f413d5cd-e143-4982-8e07-ede979564650","Type":"ContainerDied","Data":"67cf9c48a3294a238e76f09eaec91c3259ace60af98b802cdf524e6ddd6de644"}
Sep 10 16:13:18.803964 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.803840    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"
Sep 10 16:13:18.803964 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.803852    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw" event={"ID":"f413d5cd-e143-4982-8e07-ede979564650","Type":"ContainerDied","Data":"ca406d671ca7869d489148d32896f82eeda3f7093f6a1f2c5f60a88370602462"}
Sep 10 16:13:18.803964 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.803877    2564 scope.go:117] "RemoveContainer" containerID="67cf9c48a3294a238e76f09eaec91c3259ace60af98b802cdf524e6ddd6de644"
Sep 10 16:13:18.814301 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.814281    2564 scope.go:117] "RemoveContainer" containerID="7fdbe5def722097ddc2342a15f3bdf23d9e9f72c53f425d502353ccbe8f98540"
Sep 10 16:13:18.822722 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.822703    2564 scope.go:117] "RemoveContainer" containerID="67cf9c48a3294a238e76f09eaec91c3259ace60af98b802cdf524e6ddd6de644"
Sep 10 16:13:18.822996 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:13:18.822979    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67cf9c48a3294a238e76f09eaec91c3259ace60af98b802cdf524e6ddd6de644\": container with ID starting with 67cf9c48a3294a238e76f09eaec91c3259ace60af98b802cdf524e6ddd6de644 not found: ID does not exist" containerID="67cf9c48a3294a238e76f09eaec91c3259ace60af98b802cdf524e6ddd6de644"
Sep 10 16:13:18.823053 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.823005    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67cf9c48a3294a238e76f09eaec91c3259ace60af98b802cdf524e6ddd6de644"} err="failed to get container status \"67cf9c48a3294a238e76f09eaec91c3259ace60af98b802cdf524e6ddd6de644\": rpc error: code = NotFound desc = could not find container \"67cf9c48a3294a238e76f09eaec91c3259ace60af98b802cdf524e6ddd6de644\": container with ID starting with 67cf9c48a3294a238e76f09eaec91c3259ace60af98b802cdf524e6ddd6de644 not found: ID does not exist"
Sep 10 16:13:18.823053 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.823020    2564 scope.go:117] "RemoveContainer" containerID="7fdbe5def722097ddc2342a15f3bdf23d9e9f72c53f425d502353ccbe8f98540"
Sep 10 16:13:18.823289 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:13:18.823269    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fdbe5def722097ddc2342a15f3bdf23d9e9f72c53f425d502353ccbe8f98540\": container with ID starting with 7fdbe5def722097ddc2342a15f3bdf23d9e9f72c53f425d502353ccbe8f98540 not found: ID does not exist" containerID="7fdbe5def722097ddc2342a15f3bdf23d9e9f72c53f425d502353ccbe8f98540"
Sep 10 16:13:18.823345 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.823299    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fdbe5def722097ddc2342a15f3bdf23d9e9f72c53f425d502353ccbe8f98540"} err="failed to get container status \"7fdbe5def722097ddc2342a15f3bdf23d9e9f72c53f425d502353ccbe8f98540\": rpc error: code = NotFound desc = could not find container \"7fdbe5def722097ddc2342a15f3bdf23d9e9f72c53f425d502353ccbe8f98540\": container with ID starting with 7fdbe5def722097ddc2342a15f3bdf23d9e9f72c53f425d502353ccbe8f98540 not found: ID does not exist"
Sep 10 16:13:18.832231 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.832203    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"]
Sep 10 16:13:18.835672 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:18.835649    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-108dbb-openshift-default-549466454d-bjzqw"]
Sep 10 16:13:19.672814 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:13:19.672779    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f413d5cd-e143-4982-8e07-ede979564650" path="/var/lib/kubelet/pods/f413d5cd-e143-4982-8e07-ede979564650/volumes"
Sep 10 16:14:17.610555 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.610488    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cd9db79db-lrblf"]
Sep 10 16:14:17.611226 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.611198    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" containerName="istio-proxy"
Sep 10 16:14:17.611226 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.611227    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" containerName="istio-proxy"
Sep 10 16:14:17.611380 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.611238    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f413d5cd-e143-4982-8e07-ede979564650" containerName="istio-proxy"
Sep 10 16:14:17.611380 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.611246    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f413d5cd-e143-4982-8e07-ede979564650" containerName="istio-proxy"
Sep 10 16:14:17.611380 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.611265    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a167e343-148c-4d12-93fe-49287be965f2" containerName="istio-proxy"
Sep 10 16:14:17.611380 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.611272    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a167e343-148c-4d12-93fe-49287be965f2" containerName="istio-proxy"
Sep 10 16:14:17.611380 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.611319    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373a49de-5c83-48d4-a4d9-43d1b5ed6558" containerName="manager"
Sep 10 16:14:17.611380 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.611327    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="373a49de-5c83-48d4-a4d9-43d1b5ed6558" containerName="manager"
Sep 10 16:14:17.611380 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.611336    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" containerName="istio-proxy"
Sep 10 16:14:17.611380 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.611343    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" containerName="istio-proxy"
Sep 10 16:14:17.611380 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.611352    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c51b354-74f5-4893-8489-1ef7ab1f7a9a" containerName="revoke-keys"
Sep 10 16:14:17.611380 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.611359    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c51b354-74f5-4893-8489-1ef7ab1f7a9a" containerName="revoke-keys"
Sep 10 16:14:17.611380 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.611369    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f413d5cd-e143-4982-8e07-ede979564650" containerName="istio-proxy"
Sep 10 16:14:17.611380 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.611377    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f413d5cd-e143-4982-8e07-ede979564650" containerName="istio-proxy"
Sep 10 16:14:17.611788 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.611470    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c51b354-74f5-4893-8489-1ef7ab1f7a9a" containerName="revoke-keys"
Sep 10 16:14:17.611788 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.611479    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" containerName="istio-proxy"
Sep 10 16:14:17.611788 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.611485    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fe09b0b-462a-4b7a-a95b-8ed91494a7e8" containerName="istio-proxy"
Sep 10 16:14:17.611788 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.611492    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2348230d-7fa0-4046-9d47-29fc0fdabc70" containerName="istio-proxy"
Sep 10 16:14:17.611788 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.611501    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f413d5cd-e143-4982-8e07-ede979564650" containerName="istio-proxy"
Sep 10 16:14:17.611788 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.611506    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f413d5cd-e143-4982-8e07-ede979564650" containerName="istio-proxy"
Sep 10 16:14:17.611788 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.611512    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="373a49de-5c83-48d4-a4d9-43d1b5ed6558" containerName="manager"
Sep 10 16:14:17.614737 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.614722    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd9db79db-lrblf"
Sep 10 16:14:17.618261 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.618239    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 16:14:17.618388 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.618257    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 16:14:17.618388 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.618260    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 16:14:17.618388 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.618328    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wf8fd\""
Sep 10 16:14:17.625226 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.625202    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd9db79db-lrblf"]
Sep 10 16:14:17.716694 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.716658    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d83e4ea-003f-44ef-b949-703efd0c8635-webhook-cert\") pod \"maas-controller-5cd9db79db-lrblf\" (UID: \"8d83e4ea-003f-44ef-b949-703efd0c8635\") " pod="opendatahub/maas-controller-5cd9db79db-lrblf"
Sep 10 16:14:17.716905 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.716751    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv6hw\" (UniqueName: \"kubernetes.io/projected/8d83e4ea-003f-44ef-b949-703efd0c8635-kube-api-access-dv6hw\") pod \"maas-controller-5cd9db79db-lrblf\" (UID: \"8d83e4ea-003f-44ef-b949-703efd0c8635\") " pod="opendatahub/maas-controller-5cd9db79db-lrblf"
Sep 10 16:14:17.716905 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.716808    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d83e4ea-003f-44ef-b949-703efd0c8635-metrics-certs\") pod \"maas-controller-5cd9db79db-lrblf\" (UID: \"8d83e4ea-003f-44ef-b949-703efd0c8635\") " pod="opendatahub/maas-controller-5cd9db79db-lrblf"
Sep 10 16:14:17.817919 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.817884    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv6hw\" (UniqueName: \"kubernetes.io/projected/8d83e4ea-003f-44ef-b949-703efd0c8635-kube-api-access-dv6hw\") pod \"maas-controller-5cd9db79db-lrblf\" (UID: \"8d83e4ea-003f-44ef-b949-703efd0c8635\") " pod="opendatahub/maas-controller-5cd9db79db-lrblf"
Sep 10 16:14:17.818132 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.817950    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d83e4ea-003f-44ef-b949-703efd0c8635-metrics-certs\") pod \"maas-controller-5cd9db79db-lrblf\" (UID: \"8d83e4ea-003f-44ef-b949-703efd0c8635\") " pod="opendatahub/maas-controller-5cd9db79db-lrblf"
Sep 10 16:14:17.818132 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.818004    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d83e4ea-003f-44ef-b949-703efd0c8635-webhook-cert\") pod \"maas-controller-5cd9db79db-lrblf\" (UID: \"8d83e4ea-003f-44ef-b949-703efd0c8635\") " pod="opendatahub/maas-controller-5cd9db79db-lrblf"
Sep 10 16:14:17.820517 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.820496    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d83e4ea-003f-44ef-b949-703efd0c8635-metrics-certs\") pod \"maas-controller-5cd9db79db-lrblf\" (UID: \"8d83e4ea-003f-44ef-b949-703efd0c8635\") " pod="opendatahub/maas-controller-5cd9db79db-lrblf"
Sep 10 16:14:17.820618 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.820601    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d83e4ea-003f-44ef-b949-703efd0c8635-webhook-cert\") pod \"maas-controller-5cd9db79db-lrblf\" (UID: \"8d83e4ea-003f-44ef-b949-703efd0c8635\") " pod="opendatahub/maas-controller-5cd9db79db-lrblf"
Sep 10 16:14:17.827229 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.827209    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv6hw\" (UniqueName: \"kubernetes.io/projected/8d83e4ea-003f-44ef-b949-703efd0c8635-kube-api-access-dv6hw\") pod \"maas-controller-5cd9db79db-lrblf\" (UID: \"8d83e4ea-003f-44ef-b949-703efd0c8635\") " pod="opendatahub/maas-controller-5cd9db79db-lrblf"
Sep 10 16:14:17.926204 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:17.926143    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd9db79db-lrblf"
Sep 10 16:14:18.199644 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:18.199620    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd9db79db-lrblf"]
Sep 10 16:14:18.201885 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:14:18.201840    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d83e4ea_003f_44ef_b949_703efd0c8635.slice/crio-b267cf518725bbd24d4b390e05370a881fbe8e44fdc8c531b9f79226fb45171f WatchSource:0}: Error finding container b267cf518725bbd24d4b390e05370a881fbe8e44fdc8c531b9f79226fb45171f: Status 404 returned error can't find the container with id b267cf518725bbd24d4b390e05370a881fbe8e44fdc8c531b9f79226fb45171f
Sep 10 16:14:18.203259 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:18.203234    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 16:14:19.134480 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:19.134375    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd9db79db-lrblf" event={"ID":"8d83e4ea-003f-44ef-b949-703efd0c8635","Type":"ContainerStarted","Data":"e1f2ba1a0ed9e9d69e7bf162e79b407743b35b2a357a48d0f09d0a6dd6df24a8"}
Sep 10 16:14:19.134842 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:19.134532    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cd9db79db-lrblf"
Sep 10 16:14:19.134842 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:19.134557    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd9db79db-lrblf" event={"ID":"8d83e4ea-003f-44ef-b949-703efd0c8635","Type":"ContainerStarted","Data":"b267cf518725bbd24d4b390e05370a881fbe8e44fdc8c531b9f79226fb45171f"}
Sep 10 16:14:19.153195 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:19.153127    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5cd9db79db-lrblf" podStartSLOduration=1.649848332 podStartE2EDuration="2.153110405s" totalImagesPullingTime="503.262073ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:14:19.152783287 +0000 UTC m=+1937.957102551" watchObservedRunningTime="2026-09-10 16:14:19.153110405 +0000 UTC m=+1937.957429665"
Sep 10 16:14:30.143145 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:30.143109    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cd9db79db-lrblf"
Sep 10 16:14:37.330248 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:37.330208    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cd9db79db-lrblf"]
Sep 10 16:14:37.330795 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:37.330715    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5cd9db79db-lrblf" podUID="8d83e4ea-003f-44ef-b949-703efd0c8635" containerName="manager" containerID="cri-o://e1f2ba1a0ed9e9d69e7bf162e79b407743b35b2a357a48d0f09d0a6dd6df24a8" gracePeriod=30
Sep 10 16:14:37.583730 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:37.583669    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd9db79db-lrblf"
Sep 10 16:14:37.714933 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:37.714903    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv6hw\" (UniqueName: \"kubernetes.io/projected/8d83e4ea-003f-44ef-b949-703efd0c8635-kube-api-access-dv6hw\") pod \"8d83e4ea-003f-44ef-b949-703efd0c8635\" (UID: \"8d83e4ea-003f-44ef-b949-703efd0c8635\") "
Sep 10 16:14:37.715106 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:37.714940    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d83e4ea-003f-44ef-b949-703efd0c8635-metrics-certs\") pod \"8d83e4ea-003f-44ef-b949-703efd0c8635\" (UID: \"8d83e4ea-003f-44ef-b949-703efd0c8635\") "
Sep 10 16:14:37.715106 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:37.715033    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d83e4ea-003f-44ef-b949-703efd0c8635-webhook-cert\") pod \"8d83e4ea-003f-44ef-b949-703efd0c8635\" (UID: \"8d83e4ea-003f-44ef-b949-703efd0c8635\") "
Sep 10 16:14:37.717276 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:37.717251    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d83e4ea-003f-44ef-b949-703efd0c8635-kube-api-access-dv6hw" (OuterVolumeSpecName: "kube-api-access-dv6hw") pod "8d83e4ea-003f-44ef-b949-703efd0c8635" (UID: "8d83e4ea-003f-44ef-b949-703efd0c8635"). InnerVolumeSpecName "kube-api-access-dv6hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:14:37.717276 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:37.717251    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d83e4ea-003f-44ef-b949-703efd0c8635-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8d83e4ea-003f-44ef-b949-703efd0c8635" (UID: "8d83e4ea-003f-44ef-b949-703efd0c8635"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:14:37.717429 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:37.717311    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d83e4ea-003f-44ef-b949-703efd0c8635-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8d83e4ea-003f-44ef-b949-703efd0c8635" (UID: "8d83e4ea-003f-44ef-b949-703efd0c8635"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:14:37.816895 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:37.816855    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv6hw\" (UniqueName: \"kubernetes.io/projected/8d83e4ea-003f-44ef-b949-703efd0c8635-kube-api-access-dv6hw\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:14:37.816895 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:37.816887    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d83e4ea-003f-44ef-b949-703efd0c8635-metrics-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:14:37.816895 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:37.816898    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d83e4ea-003f-44ef-b949-703efd0c8635-webhook-cert\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:14:38.236924 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:38.236881    2564 generic.go:356] "Generic (PLEG): container finished" podID="8d83e4ea-003f-44ef-b949-703efd0c8635" containerID="e1f2ba1a0ed9e9d69e7bf162e79b407743b35b2a357a48d0f09d0a6dd6df24a8" exitCode=0
Sep 10 16:14:38.237116 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:38.236944    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd9db79db-lrblf"
Sep 10 16:14:38.237116 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:38.236982    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd9db79db-lrblf" event={"ID":"8d83e4ea-003f-44ef-b949-703efd0c8635","Type":"ContainerDied","Data":"e1f2ba1a0ed9e9d69e7bf162e79b407743b35b2a357a48d0f09d0a6dd6df24a8"}
Sep 10 16:14:38.237116 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:38.237050    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd9db79db-lrblf" event={"ID":"8d83e4ea-003f-44ef-b949-703efd0c8635","Type":"ContainerDied","Data":"b267cf518725bbd24d4b390e05370a881fbe8e44fdc8c531b9f79226fb45171f"}
Sep 10 16:14:38.237116 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:38.237083    2564 scope.go:117] "RemoveContainer" containerID="e1f2ba1a0ed9e9d69e7bf162e79b407743b35b2a357a48d0f09d0a6dd6df24a8"
Sep 10 16:14:38.249054 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:38.249033    2564 scope.go:117] "RemoveContainer" containerID="e1f2ba1a0ed9e9d69e7bf162e79b407743b35b2a357a48d0f09d0a6dd6df24a8"
Sep 10 16:14:38.249436 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:14:38.249406    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f2ba1a0ed9e9d69e7bf162e79b407743b35b2a357a48d0f09d0a6dd6df24a8\": container with ID starting with e1f2ba1a0ed9e9d69e7bf162e79b407743b35b2a357a48d0f09d0a6dd6df24a8 not found: ID does not exist" containerID="e1f2ba1a0ed9e9d69e7bf162e79b407743b35b2a357a48d0f09d0a6dd6df24a8"
Sep 10 16:14:38.249535 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:38.249447    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f2ba1a0ed9e9d69e7bf162e79b407743b35b2a357a48d0f09d0a6dd6df24a8"} err="failed to get container status \"e1f2ba1a0ed9e9d69e7bf162e79b407743b35b2a357a48d0f09d0a6dd6df24a8\": rpc error: code = NotFound desc = could not find container \"e1f2ba1a0ed9e9d69e7bf162e79b407743b35b2a357a48d0f09d0a6dd6df24a8\": container with ID starting with e1f2ba1a0ed9e9d69e7bf162e79b407743b35b2a357a48d0f09d0a6dd6df24a8 not found: ID does not exist"
Sep 10 16:14:38.265617 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:38.265591    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cd9db79db-lrblf"]
Sep 10 16:14:38.271540 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:38.271515    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5cd9db79db-lrblf"]
Sep 10 16:14:39.045497 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.045461    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cd9db79db-phw5r"]
Sep 10 16:14:39.045894 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.045885    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d83e4ea-003f-44ef-b949-703efd0c8635" containerName="manager"
Sep 10 16:14:39.045936 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.045897    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d83e4ea-003f-44ef-b949-703efd0c8635" containerName="manager"
Sep 10 16:14:39.045971 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.045953    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d83e4ea-003f-44ef-b949-703efd0c8635" containerName="manager"
Sep 10 16:14:39.050310 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.050289    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd9db79db-phw5r"
Sep 10 16:14:39.053207 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.053179    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 16:14:39.053879 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.053853    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 16:14:39.054006 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.053864    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wf8fd\""
Sep 10 16:14:39.054006 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.053910    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 16:14:39.060585 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.060559    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd9db79db-phw5r"]
Sep 10 16:14:39.231020 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.230984    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee1606b-16f3-4b8c-a77e-953d062c69e4-metrics-certs\") pod \"maas-controller-5cd9db79db-phw5r\" (UID: \"5ee1606b-16f3-4b8c-a77e-953d062c69e4\") " pod="opendatahub/maas-controller-5cd9db79db-phw5r"
Sep 10 16:14:39.231020 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.231019    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b8n4\" (UniqueName: \"kubernetes.io/projected/5ee1606b-16f3-4b8c-a77e-953d062c69e4-kube-api-access-9b8n4\") pod \"maas-controller-5cd9db79db-phw5r\" (UID: \"5ee1606b-16f3-4b8c-a77e-953d062c69e4\") " pod="opendatahub/maas-controller-5cd9db79db-phw5r"
Sep 10 16:14:39.231310 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.231044    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ee1606b-16f3-4b8c-a77e-953d062c69e4-webhook-cert\") pod \"maas-controller-5cd9db79db-phw5r\" (UID: \"5ee1606b-16f3-4b8c-a77e-953d062c69e4\") " pod="opendatahub/maas-controller-5cd9db79db-phw5r"
Sep 10 16:14:39.332327 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.332241    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee1606b-16f3-4b8c-a77e-953d062c69e4-metrics-certs\") pod \"maas-controller-5cd9db79db-phw5r\" (UID: \"5ee1606b-16f3-4b8c-a77e-953d062c69e4\") " pod="opendatahub/maas-controller-5cd9db79db-phw5r"
Sep 10 16:14:39.332327 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.332278    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b8n4\" (UniqueName: \"kubernetes.io/projected/5ee1606b-16f3-4b8c-a77e-953d062c69e4-kube-api-access-9b8n4\") pod \"maas-controller-5cd9db79db-phw5r\" (UID: \"5ee1606b-16f3-4b8c-a77e-953d062c69e4\") " pod="opendatahub/maas-controller-5cd9db79db-phw5r"
Sep 10 16:14:39.332327 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.332302    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ee1606b-16f3-4b8c-a77e-953d062c69e4-webhook-cert\") pod \"maas-controller-5cd9db79db-phw5r\" (UID: \"5ee1606b-16f3-4b8c-a77e-953d062c69e4\") " pod="opendatahub/maas-controller-5cd9db79db-phw5r"
Sep 10 16:14:39.335028 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.335003    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee1606b-16f3-4b8c-a77e-953d062c69e4-metrics-certs\") pod \"maas-controller-5cd9db79db-phw5r\" (UID: \"5ee1606b-16f3-4b8c-a77e-953d062c69e4\") " pod="opendatahub/maas-controller-5cd9db79db-phw5r"
Sep 10 16:14:39.335148 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.335032    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5ee1606b-16f3-4b8c-a77e-953d062c69e4-webhook-cert\") pod \"maas-controller-5cd9db79db-phw5r\" (UID: \"5ee1606b-16f3-4b8c-a77e-953d062c69e4\") " pod="opendatahub/maas-controller-5cd9db79db-phw5r"
Sep 10 16:14:39.341993 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.341967    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b8n4\" (UniqueName: \"kubernetes.io/projected/5ee1606b-16f3-4b8c-a77e-953d062c69e4-kube-api-access-9b8n4\") pod \"maas-controller-5cd9db79db-phw5r\" (UID: \"5ee1606b-16f3-4b8c-a77e-953d062c69e4\") " pod="opendatahub/maas-controller-5cd9db79db-phw5r"
Sep 10 16:14:39.361832 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.361800    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd9db79db-phw5r"
Sep 10 16:14:39.634804 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.634766    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd9db79db-phw5r"]
Sep 10 16:14:39.637066 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:14:39.637040    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee1606b_16f3_4b8c_a77e_953d062c69e4.slice/crio-ae663813dc17795bb5475fb54fd2765673f5012eb514ebb29bafcc558124b8ab WatchSource:0}: Error finding container ae663813dc17795bb5475fb54fd2765673f5012eb514ebb29bafcc558124b8ab: Status 404 returned error can't find the container with id ae663813dc17795bb5475fb54fd2765673f5012eb514ebb29bafcc558124b8ab
Sep 10 16:14:39.673357 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:39.673327    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d83e4ea-003f-44ef-b949-703efd0c8635" path="/var/lib/kubelet/pods/8d83e4ea-003f-44ef-b949-703efd0c8635/volumes"
Sep 10 16:14:40.250293 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:40.250255    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd9db79db-phw5r" event={"ID":"5ee1606b-16f3-4b8c-a77e-953d062c69e4","Type":"ContainerStarted","Data":"af9b250d1bd31c2295151227315524f0c938beae26b9d927e5fa5dac5b465e65"}
Sep 10 16:14:40.250691 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:40.250310    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd9db79db-phw5r" event={"ID":"5ee1606b-16f3-4b8c-a77e-953d062c69e4","Type":"ContainerStarted","Data":"ae663813dc17795bb5475fb54fd2765673f5012eb514ebb29bafcc558124b8ab"}
Sep 10 16:14:40.250691 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:40.250438    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cd9db79db-phw5r"
Sep 10 16:14:40.273216 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:40.273146    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5cd9db79db-phw5r" podStartSLOduration=0.805606173 podStartE2EDuration="1.273130415s" totalImagesPullingTime="467.524242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:14:40.271397232 +0000 UTC m=+1959.075716505" watchObservedRunningTime="2026-09-10 16:14:40.273130415 +0000 UTC m=+1959.077449676"
Sep 10 16:14:51.259048 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:14:51.259015    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cd9db79db-phw5r"
Sep 10 16:15:00.134104 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:00.134021    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817615-frx7b"]
Sep 10 16:15:00.137972 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:00.137943    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817615-frx7b"
Sep 10 16:15:00.140355 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:00.140334    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2wghg\""
Sep 10 16:15:00.144844 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:00.144823    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817615-frx7b"]
Sep 10 16:15:00.151775 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:00.151749    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817615-frx7b"
Sep 10 16:15:00.426672 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:00.426645    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817615-frx7b"]
Sep 10 16:15:00.429127 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:15:00.429103    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b3ef8b6_9a8e_4d75_b609_dc2df71220cf.slice/crio-2c8c61253ccfd54bcac19b770c66a47dfad812ec9ddff8236922b56566891a05 WatchSource:0}: Error finding container 2c8c61253ccfd54bcac19b770c66a47dfad812ec9ddff8236922b56566891a05: Status 404 returned error can't find the container with id 2c8c61253ccfd54bcac19b770c66a47dfad812ec9ddff8236922b56566891a05
Sep 10 16:15:01.365032 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:01.364989    2564 generic.go:356] "Generic (PLEG): container finished" podID="2b3ef8b6-9a8e-4d75-b609-dc2df71220cf" containerID="c2097618a193519bd7a0b53d1e817fe9654fb1abb73fbf8497377b86e25825d4" exitCode=0
Sep 10 16:15:01.365032 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:01.365035    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817615-frx7b" event={"ID":"2b3ef8b6-9a8e-4d75-b609-dc2df71220cf","Type":"ContainerDied","Data":"c2097618a193519bd7a0b53d1e817fe9654fb1abb73fbf8497377b86e25825d4"}
Sep 10 16:15:01.365479 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:01.365082    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817615-frx7b" event={"ID":"2b3ef8b6-9a8e-4d75-b609-dc2df71220cf","Type":"ContainerStarted","Data":"2c8c61253ccfd54bcac19b770c66a47dfad812ec9ddff8236922b56566891a05"}
Sep 10 16:15:02.509052 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:02.509027    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817615-frx7b"
Sep 10 16:15:03.377372 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:03.377342    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817615-frx7b"
Sep 10 16:15:03.377372 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:03.377363    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817615-frx7b" event={"ID":"2b3ef8b6-9a8e-4d75-b609-dc2df71220cf","Type":"ContainerDied","Data":"2c8c61253ccfd54bcac19b770c66a47dfad812ec9ddff8236922b56566891a05"}
Sep 10 16:15:03.377640 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:03.377408    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c8c61253ccfd54bcac19b770c66a47dfad812ec9ddff8236922b56566891a05"
Sep 10 16:15:56.472230 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:56.472197    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx"]
Sep 10 16:15:56.472753 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:56.472404    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx" podUID="a90ce5c4-9824-4886-9f8c-caacf5555176" containerName="manager" containerID="cri-o://9f8193d994ab74279e3f895e9c80c3d7d9c059329807bb07155ae814dc7c1607" gracePeriod=10
Sep 10 16:15:56.827374 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:56.827346    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx"
Sep 10 16:15:56.959774 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:56.959739    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a90ce5c4-9824-4886-9f8c-caacf5555176-extensions-socket-volume\") pod \"a90ce5c4-9824-4886-9f8c-caacf5555176\" (UID: \"a90ce5c4-9824-4886-9f8c-caacf5555176\") "
Sep 10 16:15:56.959774 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:56.959777    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6l8x\" (UniqueName: \"kubernetes.io/projected/a90ce5c4-9824-4886-9f8c-caacf5555176-kube-api-access-b6l8x\") pod \"a90ce5c4-9824-4886-9f8c-caacf5555176\" (UID: \"a90ce5c4-9824-4886-9f8c-caacf5555176\") "
Sep 10 16:15:56.960126 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:56.960103    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a90ce5c4-9824-4886-9f8c-caacf5555176-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a90ce5c4-9824-4886-9f8c-caacf5555176" (UID: "a90ce5c4-9824-4886-9f8c-caacf5555176"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:15:56.962082 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:56.962063    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a90ce5c4-9824-4886-9f8c-caacf5555176-kube-api-access-b6l8x" (OuterVolumeSpecName: "kube-api-access-b6l8x") pod "a90ce5c4-9824-4886-9f8c-caacf5555176" (UID: "a90ce5c4-9824-4886-9f8c-caacf5555176"). InnerVolumeSpecName "kube-api-access-b6l8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:15:57.060706 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:57.060614    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6l8x\" (UniqueName: \"kubernetes.io/projected/a90ce5c4-9824-4886-9f8c-caacf5555176-kube-api-access-b6l8x\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:15:57.060706 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:57.060649    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a90ce5c4-9824-4886-9f8c-caacf5555176-extensions-socket-volume\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:15:57.679942 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:57.679903    2564 generic.go:356] "Generic (PLEG): container finished" podID="a90ce5c4-9824-4886-9f8c-caacf5555176" containerID="9f8193d994ab74279e3f895e9c80c3d7d9c059329807bb07155ae814dc7c1607" exitCode=0
Sep 10 16:15:57.680437 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:57.679977    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx"
Sep 10 16:15:57.680437 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:57.679996    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx" event={"ID":"a90ce5c4-9824-4886-9f8c-caacf5555176","Type":"ContainerDied","Data":"9f8193d994ab74279e3f895e9c80c3d7d9c059329807bb07155ae814dc7c1607"}
Sep 10 16:15:57.680437 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:57.680059    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx" event={"ID":"a90ce5c4-9824-4886-9f8c-caacf5555176","Type":"ContainerDied","Data":"d768a78d9148ba1e362727f7e60879a8e48de502216b67de51190546a1c1b58f"}
Sep 10 16:15:57.680437 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:57.680093    2564 scope.go:117] "RemoveContainer" containerID="9f8193d994ab74279e3f895e9c80c3d7d9c059329807bb07155ae814dc7c1607"
Sep 10 16:15:57.691492 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:57.690658    2564 scope.go:117] "RemoveContainer" containerID="9f8193d994ab74279e3f895e9c80c3d7d9c059329807bb07155ae814dc7c1607"
Sep 10 16:15:57.696807 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:15:57.696776    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f8193d994ab74279e3f895e9c80c3d7d9c059329807bb07155ae814dc7c1607\": container with ID starting with 9f8193d994ab74279e3f895e9c80c3d7d9c059329807bb07155ae814dc7c1607 not found: ID does not exist" containerID="9f8193d994ab74279e3f895e9c80c3d7d9c059329807bb07155ae814dc7c1607"
Sep 10 16:15:57.696920 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:57.696830    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f8193d994ab74279e3f895e9c80c3d7d9c059329807bb07155ae814dc7c1607"} err="failed to get container status \"9f8193d994ab74279e3f895e9c80c3d7d9c059329807bb07155ae814dc7c1607\": rpc error: code = NotFound desc = could not find container \"9f8193d994ab74279e3f895e9c80c3d7d9c059329807bb07155ae814dc7c1607\": container with ID starting with 9f8193d994ab74279e3f895e9c80c3d7d9c059329807bb07155ae814dc7c1607 not found: ID does not exist"
Sep 10 16:15:57.702660 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:57.702636    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx"]
Sep 10 16:15:57.708329 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:57.708309    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-47bpx"]
Sep 10 16:15:59.672849 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:15:59.672814    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a90ce5c4-9824-4886-9f8c-caacf5555176" path="/var/lib/kubelet/pods/a90ce5c4-9824-4886-9f8c-caacf5555176/volumes"
Sep 10 16:16:57.185594 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:57.185554    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8wmh9"]
Sep 10 16:16:57.186074 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:57.185974    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a90ce5c4-9824-4886-9f8c-caacf5555176" containerName="manager"
Sep 10 16:16:57.186074 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:57.185986    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a90ce5c4-9824-4886-9f8c-caacf5555176" containerName="manager"
Sep 10 16:16:57.186074 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:57.186012    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b3ef8b6-9a8e-4d75-b609-dc2df71220cf" containerName="cleanup"
Sep 10 16:16:57.186074 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:57.186017    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b3ef8b6-9a8e-4d75-b609-dc2df71220cf" containerName="cleanup"
Sep 10 16:16:57.186257 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:57.186099    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b3ef8b6-9a8e-4d75-b609-dc2df71220cf" containerName="cleanup"
Sep 10 16:16:57.186257 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:57.186108    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a90ce5c4-9824-4886-9f8c-caacf5555176" containerName="manager"
Sep 10 16:16:57.189364 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:57.189348    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8wmh9"
Sep 10 16:16:57.192009 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:57.191989    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-w9tpd\""
Sep 10 16:16:57.204038 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:57.204015    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8wmh9"]
Sep 10 16:16:57.234085 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:57.234046    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn9ph\" (UniqueName: \"kubernetes.io/projected/bfc5bc5c-d9b9-40fb-b6a7-50711c9bcfb8-kube-api-access-vn9ph\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8wmh9\" (UID: \"bfc5bc5c-d9b9-40fb-b6a7-50711c9bcfb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8wmh9"
Sep 10 16:16:57.234281 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:57.234096    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfc5bc5c-d9b9-40fb-b6a7-50711c9bcfb8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8wmh9\" (UID: \"bfc5bc5c-d9b9-40fb-b6a7-50711c9bcfb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8wmh9"
Sep 10 16:16:57.335342 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:57.335299    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn9ph\" (UniqueName: \"kubernetes.io/projected/bfc5bc5c-d9b9-40fb-b6a7-50711c9bcfb8-kube-api-access-vn9ph\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8wmh9\" (UID: \"bfc5bc5c-d9b9-40fb-b6a7-50711c9bcfb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8wmh9"
Sep 10 16:16:57.335533 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:57.335361    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfc5bc5c-d9b9-40fb-b6a7-50711c9bcfb8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8wmh9\" (UID: \"bfc5bc5c-d9b9-40fb-b6a7-50711c9bcfb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8wmh9"
Sep 10 16:16:57.335725 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:57.335708    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfc5bc5c-d9b9-40fb-b6a7-50711c9bcfb8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8wmh9\" (UID: \"bfc5bc5c-d9b9-40fb-b6a7-50711c9bcfb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8wmh9"
Sep 10 16:16:57.346012 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:57.345987    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn9ph\" (UniqueName: \"kubernetes.io/projected/bfc5bc5c-d9b9-40fb-b6a7-50711c9bcfb8-kube-api-access-vn9ph\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8wmh9\" (UID: \"bfc5bc5c-d9b9-40fb-b6a7-50711c9bcfb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8wmh9"
Sep 10 16:16:57.499553 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:57.499458    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8wmh9"
Sep 10 16:16:57.779607 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:57.779579    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8wmh9"]
Sep 10 16:16:57.782587 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:16:57.782556    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc5bc5c_d9b9_40fb_b6a7_50711c9bcfb8.slice/crio-a87c8b88f9a7893e164715ab78a3dfd5cf373b589a0114a1b72ab908efc473bf WatchSource:0}: Error finding container a87c8b88f9a7893e164715ab78a3dfd5cf373b589a0114a1b72ab908efc473bf: Status 404 returned error can't find the container with id a87c8b88f9a7893e164715ab78a3dfd5cf373b589a0114a1b72ab908efc473bf
Sep 10 16:16:57.995114 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:57.995076    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8wmh9" event={"ID":"bfc5bc5c-d9b9-40fb-b6a7-50711c9bcfb8","Type":"ContainerStarted","Data":"8a9686c9bfd2a20f4fc2d4ca81e8c69ddf0f9dbf22a7430f684ffe704475917e"}
Sep 10 16:16:57.995352 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:57.995137    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8wmh9" event={"ID":"bfc5bc5c-d9b9-40fb-b6a7-50711c9bcfb8","Type":"ContainerStarted","Data":"a87c8b88f9a7893e164715ab78a3dfd5cf373b589a0114a1b72ab908efc473bf"}
Sep 10 16:16:57.995352 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:57.995201    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8wmh9"
Sep 10 16:16:58.022064 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:16:58.021994    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8wmh9" podStartSLOduration=1.021975506 podStartE2EDuration="1.021975506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:16:58.021171467 +0000 UTC m=+2096.825490721" watchObservedRunningTime="2026-09-10 16:16:58.021975506 +0000 UTC m=+2096.826294766"
Sep 10 16:17:02.524854 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:17:02.524818    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-snm4z_f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f/console-operator/1.log"
Sep 10 16:17:02.528994 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:17:02.528973    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-snm4z_f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f/console-operator/1.log"
Sep 10 16:17:09.003875 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:17:09.003844    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8wmh9"
Sep 10 16:18:01.372672 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.372581    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"]
Sep 10 16:18:01.376550 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.376526    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:01.378986 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.378964    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-44b3e5ce-kserve-self-signed-certs\""
Sep 10 16:18:01.405056 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.405034    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"]
Sep 10 16:18:01.472995 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.472960    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-dshm\") pod \"e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") " pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:01.472995 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.472998    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28dwx\" (UniqueName: \"kubernetes.io/projected/bc74959a-5c23-40dd-b553-56d848e52614-kube-api-access-28dwx\") pod \"e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") " pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:01.473235 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.473029    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-tmp-dir\") pod \"e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") " pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:01.473235 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.473078    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc74959a-5c23-40dd-b553-56d848e52614-tls-certs\") pod \"e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") " pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:01.473235 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.473130    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-home\") pod \"e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") " pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:01.473235 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.473185    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-model-cache\") pod \"e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") " pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:01.574034 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.573995    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-home\") pod \"e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") " pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:01.574230 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.574049    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-model-cache\") pod \"e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") " pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:01.574230 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.574127    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-dshm\") pod \"e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") " pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:01.574230 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.574152    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28dwx\" (UniqueName: \"kubernetes.io/projected/bc74959a-5c23-40dd-b553-56d848e52614-kube-api-access-28dwx\") pod \"e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") " pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:01.574393 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.574235    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-tmp-dir\") pod \"e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") " pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:01.574393 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.574293    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc74959a-5c23-40dd-b553-56d848e52614-tls-certs\") pod \"e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") " pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:01.574507 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.574485    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-home\") pod \"e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") " pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:01.574563 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.574544    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-model-cache\") pod \"e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") " pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:01.574831 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.574802    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-tmp-dir\") pod \"e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") " pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:01.577110 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.577086    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-44b3e5ce-kserve-self-signed-certs\""
Sep 10 16:18:01.577255 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.577109    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-dshm\") pod \"e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") " pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:01.586101 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.586075    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28dwx\" (UniqueName: \"kubernetes.io/projected/bc74959a-5c23-40dd-b553-56d848e52614-kube-api-access-28dwx\") pod \"e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") " pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:01.587186 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.587147    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc74959a-5c23-40dd-b553-56d848e52614-tls-certs\") pod \"e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") " pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:01.685406 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.685374    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:01.967985 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:01.967963    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"]
Sep 10 16:18:01.969944 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:18:01.969918    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc74959a_5c23_40dd_b553_56d848e52614.slice/crio-54dae6e8225d7c2e2bfacbe69dc755e43d8ccb056d2e1480e9f06bc02f6ac177 WatchSource:0}: Error finding container 54dae6e8225d7c2e2bfacbe69dc755e43d8ccb056d2e1480e9f06bc02f6ac177: Status 404 returned error can't find the container with id 54dae6e8225d7c2e2bfacbe69dc755e43d8ccb056d2e1480e9f06bc02f6ac177
Sep 10 16:18:02.361613 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:02.361517    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69" event={"ID":"bc74959a-5c23-40dd-b553-56d848e52614","Type":"ContainerStarted","Data":"3409a148f028448ea1f87846a33f538b0ad18667ee7792c8a45f04b7cb9c634c"}
Sep 10 16:18:02.361613 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:02.361572    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69" event={"ID":"bc74959a-5c23-40dd-b553-56d848e52614","Type":"ContainerStarted","Data":"54dae6e8225d7c2e2bfacbe69dc755e43d8ccb056d2e1480e9f06bc02f6ac177"}
Sep 10 16:18:02.382919 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:02.382869    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69" podStartSLOduration=1.382855677 podStartE2EDuration="1.382855677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:02.381483741 +0000 UTC m=+2161.185803015" watchObservedRunningTime="2026-09-10 16:18:02.382855677 +0000 UTC m=+2161.187174937"
Sep 10 16:18:11.686060 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:11.686024    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:11.686758 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:11.686073    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:11.699504 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:11.699479    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:12.434521 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:12.434491    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:26.040515 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:26.040481    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"]
Sep 10 16:18:41.060726 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.060684    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69" podUID="bc74959a-5c23-40dd-b553-56d848e52614" containerName="main" containerID="cri-o://3409a148f028448ea1f87846a33f538b0ad18667ee7792c8a45f04b7cb9c634c" gracePeriod=45
Sep 10 16:18:41.320509 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.320453    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:41.470861 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.470835    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-dshm\") pod \"bc74959a-5c23-40dd-b553-56d848e52614\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") "
Sep 10 16:18:41.470861 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.470868    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-home\") pod \"bc74959a-5c23-40dd-b553-56d848e52614\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") "
Sep 10 16:18:41.471111 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.470893    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28dwx\" (UniqueName: \"kubernetes.io/projected/bc74959a-5c23-40dd-b553-56d848e52614-kube-api-access-28dwx\") pod \"bc74959a-5c23-40dd-b553-56d848e52614\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") "
Sep 10 16:18:41.471111 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.470940    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-tmp-dir\") pod \"bc74959a-5c23-40dd-b553-56d848e52614\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") "
Sep 10 16:18:41.471111 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.470963    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-model-cache\") pod \"bc74959a-5c23-40dd-b553-56d848e52614\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") "
Sep 10 16:18:41.471111 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.471003    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc74959a-5c23-40dd-b553-56d848e52614-tls-certs\") pod \"bc74959a-5c23-40dd-b553-56d848e52614\" (UID: \"bc74959a-5c23-40dd-b553-56d848e52614\") "
Sep 10 16:18:41.471463 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.471201    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-home" (OuterVolumeSpecName: "home") pod "bc74959a-5c23-40dd-b553-56d848e52614" (UID: "bc74959a-5c23-40dd-b553-56d848e52614"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:18:41.471463 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.471262    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc74959a-5c23-40dd-b553-56d848e52614" (UID: "bc74959a-5c23-40dd-b553-56d848e52614"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:18:41.471463 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.471346    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc74959a-5c23-40dd-b553-56d848e52614" (UID: "bc74959a-5c23-40dd-b553-56d848e52614"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:18:41.471463 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.471398    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-home\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:18:41.471463 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.471411    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-tmp-dir\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:18:41.473186 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.473143    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-dshm" (OuterVolumeSpecName: "dshm") pod "bc74959a-5c23-40dd-b553-56d848e52614" (UID: "bc74959a-5c23-40dd-b553-56d848e52614"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:18:41.473305 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.473237    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc74959a-5c23-40dd-b553-56d848e52614-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc74959a-5c23-40dd-b553-56d848e52614" (UID: "bc74959a-5c23-40dd-b553-56d848e52614"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:18:41.473388 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.473366    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc74959a-5c23-40dd-b553-56d848e52614-kube-api-access-28dwx" (OuterVolumeSpecName: "kube-api-access-28dwx") pod "bc74959a-5c23-40dd-b553-56d848e52614" (UID: "bc74959a-5c23-40dd-b553-56d848e52614"). InnerVolumeSpecName "kube-api-access-28dwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:18:41.572723 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.572631    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-model-cache\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:18:41.572723 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.572660    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc74959a-5c23-40dd-b553-56d848e52614-tls-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:18:41.572723 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.572670    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc74959a-5c23-40dd-b553-56d848e52614-dshm\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:18:41.572723 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.572678    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28dwx\" (UniqueName: \"kubernetes.io/projected/bc74959a-5c23-40dd-b553-56d848e52614-kube-api-access-28dwx\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:18:41.580563 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.579072    2564 generic.go:356] "Generic (PLEG): container finished" podID="bc74959a-5c23-40dd-b553-56d848e52614" containerID="3409a148f028448ea1f87846a33f538b0ad18667ee7792c8a45f04b7cb9c634c" exitCode=0
Sep 10 16:18:41.580563 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.579287    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69" event={"ID":"bc74959a-5c23-40dd-b553-56d848e52614","Type":"ContainerDied","Data":"3409a148f028448ea1f87846a33f538b0ad18667ee7792c8a45f04b7cb9c634c"}
Sep 10 16:18:41.580563 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.579328    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69" event={"ID":"bc74959a-5c23-40dd-b553-56d848e52614","Type":"ContainerDied","Data":"54dae6e8225d7c2e2bfacbe69dc755e43d8ccb056d2e1480e9f06bc02f6ac177"}
Sep 10 16:18:41.580563 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.579363    2564 scope.go:117] "RemoveContainer" containerID="3409a148f028448ea1f87846a33f538b0ad18667ee7792c8a45f04b7cb9c634c"
Sep 10 16:18:41.580563 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.579559    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"
Sep 10 16:18:41.595738 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.595715    2564 scope.go:117] "RemoveContainer" containerID="3409a148f028448ea1f87846a33f538b0ad18667ee7792c8a45f04b7cb9c634c"
Sep 10 16:18:41.596053 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:18:41.596034    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3409a148f028448ea1f87846a33f538b0ad18667ee7792c8a45f04b7cb9c634c\": container with ID starting with 3409a148f028448ea1f87846a33f538b0ad18667ee7792c8a45f04b7cb9c634c not found: ID does not exist" containerID="3409a148f028448ea1f87846a33f538b0ad18667ee7792c8a45f04b7cb9c634c"
Sep 10 16:18:41.596108 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.596063    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3409a148f028448ea1f87846a33f538b0ad18667ee7792c8a45f04b7cb9c634c"} err="failed to get container status \"3409a148f028448ea1f87846a33f538b0ad18667ee7792c8a45f04b7cb9c634c\": rpc error: code = NotFound desc = could not find container \"3409a148f028448ea1f87846a33f538b0ad18667ee7792c8a45f04b7cb9c634c\": container with ID starting with 3409a148f028448ea1f87846a33f538b0ad18667ee7792c8a45f04b7cb9c634c not found: ID does not exist"
Sep 10 16:18:41.609622 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.609595    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"]
Sep 10 16:18:41.614479 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.614444    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-44b3e5ce-kserve-56cc776f59-pkm69"]
Sep 10 16:18:41.672695 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:41.672664    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc74959a-5c23-40dd-b553-56d848e52614" path="/var/lib/kubelet/pods/bc74959a-5c23-40dd-b553-56d848e52614/volumes"
Sep 10 16:18:45.093300 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.093261    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"]
Sep 10 16:18:45.093749 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.093733    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc74959a-5c23-40dd-b553-56d848e52614" containerName="main"
Sep 10 16:18:45.093802 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.093752    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc74959a-5c23-40dd-b553-56d848e52614" containerName="main"
Sep 10 16:18:45.093837 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.093816    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc74959a-5c23-40dd-b553-56d848e52614" containerName="main"
Sep 10 16:18:45.098343 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.098312    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"
Sep 10 16:18:45.102129 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.102105    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 16:18:45.102282 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.102198    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 16:18:45.102282 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.102231    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qfdsq\""
Sep 10 16:18:45.113702 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.113673    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"]
Sep 10 16:18:45.207469 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.207434    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5a087a-11b0-4580-b3a2-2aef32e51516-openshift-service-ca\") pod \"maas-api-7f5df8d666-p6l87\" (UID: \"9e5a087a-11b0-4580-b3a2-2aef32e51516\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"
Sep 10 16:18:45.207646 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.207476    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjhvp\" (UniqueName: \"kubernetes.io/projected/9e5a087a-11b0-4580-b3a2-2aef32e51516-kube-api-access-qjhvp\") pod \"maas-api-7f5df8d666-p6l87\" (UID: \"9e5a087a-11b0-4580-b3a2-2aef32e51516\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"
Sep 10 16:18:45.207646 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.207506    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5a087a-11b0-4580-b3a2-2aef32e51516-maas-api-tls\") pod \"maas-api-7f5df8d666-p6l87\" (UID: \"9e5a087a-11b0-4580-b3a2-2aef32e51516\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"
Sep 10 16:18:45.207646 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.207545    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e5a087a-11b0-4580-b3a2-2aef32e51516-metrics-certs\") pod \"maas-api-7f5df8d666-p6l87\" (UID: \"9e5a087a-11b0-4580-b3a2-2aef32e51516\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"
Sep 10 16:18:45.308566 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.308532    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5a087a-11b0-4580-b3a2-2aef32e51516-openshift-service-ca\") pod \"maas-api-7f5df8d666-p6l87\" (UID: \"9e5a087a-11b0-4580-b3a2-2aef32e51516\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"
Sep 10 16:18:45.308808 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.308586    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjhvp\" (UniqueName: \"kubernetes.io/projected/9e5a087a-11b0-4580-b3a2-2aef32e51516-kube-api-access-qjhvp\") pod \"maas-api-7f5df8d666-p6l87\" (UID: \"9e5a087a-11b0-4580-b3a2-2aef32e51516\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"
Sep 10 16:18:45.308808 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.308609    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5a087a-11b0-4580-b3a2-2aef32e51516-maas-api-tls\") pod \"maas-api-7f5df8d666-p6l87\" (UID: \"9e5a087a-11b0-4580-b3a2-2aef32e51516\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"
Sep 10 16:18:45.308808 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.308660    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e5a087a-11b0-4580-b3a2-2aef32e51516-metrics-certs\") pod \"maas-api-7f5df8d666-p6l87\" (UID: \"9e5a087a-11b0-4580-b3a2-2aef32e51516\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"
Sep 10 16:18:45.309456 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.309399    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5a087a-11b0-4580-b3a2-2aef32e51516-openshift-service-ca\") pod \"maas-api-7f5df8d666-p6l87\" (UID: \"9e5a087a-11b0-4580-b3a2-2aef32e51516\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"
Sep 10 16:18:45.311687 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.311660    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5a087a-11b0-4580-b3a2-2aef32e51516-maas-api-tls\") pod \"maas-api-7f5df8d666-p6l87\" (UID: \"9e5a087a-11b0-4580-b3a2-2aef32e51516\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"
Sep 10 16:18:45.311777 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.311667    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e5a087a-11b0-4580-b3a2-2aef32e51516-metrics-certs\") pod \"maas-api-7f5df8d666-p6l87\" (UID: \"9e5a087a-11b0-4580-b3a2-2aef32e51516\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"
Sep 10 16:18:45.319761 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.319737    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjhvp\" (UniqueName: \"kubernetes.io/projected/9e5a087a-11b0-4580-b3a2-2aef32e51516-kube-api-access-qjhvp\") pod \"maas-api-7f5df8d666-p6l87\" (UID: \"9e5a087a-11b0-4580-b3a2-2aef32e51516\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"
Sep 10 16:18:45.412385 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.412294    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"
Sep 10 16:18:45.721121 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:45.721035    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"]
Sep 10 16:18:46.613636 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:46.613537    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87" event={"ID":"9e5a087a-11b0-4580-b3a2-2aef32e51516","Type":"ContainerStarted","Data":"3b1c7956d5e4f6ada430e50a8dcce5f1804bde034660d631ab8ac9297c1c4aa1"}
Sep 10 16:18:46.613636 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:46.613599    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"
Sep 10 16:18:46.613636 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:46.613615    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87" event={"ID":"9e5a087a-11b0-4580-b3a2-2aef32e51516","Type":"ContainerStarted","Data":"faa88ae60282c481b19ea46a9fdc877d3cf31e933baa16a8b1a4ba15e2f830e9"}
Sep 10 16:18:46.634019 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:46.633959    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87" podStartSLOduration=1.127692937 podStartE2EDuration="1.633942895s" totalImagesPullingTime="506.249958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:46.63179217 +0000 UTC m=+2205.436111456" watchObservedRunningTime="2026-09-10 16:18:46.633942895 +0000 UTC m=+2205.438262156"
Sep 10 16:18:52.625060 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:52.625033    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"
Sep 10 16:18:52.916618 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:52.916535    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-68786f5998-stt8t"]
Sep 10 16:18:52.919953 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:52.919931    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68786f5998-stt8t"
Sep 10 16:18:52.933817 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:52.933765    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68786f5998-stt8t"]
Sep 10 16:18:52.999647 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:52.999611    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrb6g\" (UniqueName: \"kubernetes.io/projected/f9fe8919-0e96-44f7-adfe-1555e8be9959-kube-api-access-qrb6g\") pod \"maas-api-68786f5998-stt8t\" (UID: \"f9fe8919-0e96-44f7-adfe-1555e8be9959\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-stt8t"
Sep 10 16:18:52.999847 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:52.999696    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9fe8919-0e96-44f7-adfe-1555e8be9959-openshift-service-ca\") pod \"maas-api-68786f5998-stt8t\" (UID: \"f9fe8919-0e96-44f7-adfe-1555e8be9959\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-stt8t"
Sep 10 16:18:52.999847 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:52.999724    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9fe8919-0e96-44f7-adfe-1555e8be9959-metrics-certs\") pod \"maas-api-68786f5998-stt8t\" (UID: \"f9fe8919-0e96-44f7-adfe-1555e8be9959\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-stt8t"
Sep 10 16:18:52.999847 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:52.999817    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9fe8919-0e96-44f7-adfe-1555e8be9959-maas-api-tls\") pod \"maas-api-68786f5998-stt8t\" (UID: \"f9fe8919-0e96-44f7-adfe-1555e8be9959\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-stt8t"
Sep 10 16:18:53.101262 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:53.101224    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9fe8919-0e96-44f7-adfe-1555e8be9959-openshift-service-ca\") pod \"maas-api-68786f5998-stt8t\" (UID: \"f9fe8919-0e96-44f7-adfe-1555e8be9959\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-stt8t"
Sep 10 16:18:53.101472 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:53.101274    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9fe8919-0e96-44f7-adfe-1555e8be9959-metrics-certs\") pod \"maas-api-68786f5998-stt8t\" (UID: \"f9fe8919-0e96-44f7-adfe-1555e8be9959\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-stt8t"
Sep 10 16:18:53.101472 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:53.101427    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9fe8919-0e96-44f7-adfe-1555e8be9959-maas-api-tls\") pod \"maas-api-68786f5998-stt8t\" (UID: \"f9fe8919-0e96-44f7-adfe-1555e8be9959\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-stt8t"
Sep 10 16:18:53.101608 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:53.101533    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrb6g\" (UniqueName: \"kubernetes.io/projected/f9fe8919-0e96-44f7-adfe-1555e8be9959-kube-api-access-qrb6g\") pod \"maas-api-68786f5998-stt8t\" (UID: \"f9fe8919-0e96-44f7-adfe-1555e8be9959\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-stt8t"
Sep 10 16:18:53.102122 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:53.102039    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9fe8919-0e96-44f7-adfe-1555e8be9959-openshift-service-ca\") pod \"maas-api-68786f5998-stt8t\" (UID: \"f9fe8919-0e96-44f7-adfe-1555e8be9959\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-stt8t"
Sep 10 16:18:53.104101 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:53.104073    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9fe8919-0e96-44f7-adfe-1555e8be9959-metrics-certs\") pod \"maas-api-68786f5998-stt8t\" (UID: \"f9fe8919-0e96-44f7-adfe-1555e8be9959\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-stt8t"
Sep 10 16:18:53.104230 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:53.104073    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9fe8919-0e96-44f7-adfe-1555e8be9959-maas-api-tls\") pod \"maas-api-68786f5998-stt8t\" (UID: \"f9fe8919-0e96-44f7-adfe-1555e8be9959\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-stt8t"
Sep 10 16:18:53.111567 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:53.111539    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrb6g\" (UniqueName: \"kubernetes.io/projected/f9fe8919-0e96-44f7-adfe-1555e8be9959-kube-api-access-qrb6g\") pod \"maas-api-68786f5998-stt8t\" (UID: \"f9fe8919-0e96-44f7-adfe-1555e8be9959\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-stt8t"
Sep 10 16:18:53.233658 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:53.233613    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68786f5998-stt8t"
Sep 10 16:18:53.509788 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:53.509763    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68786f5998-stt8t"]
Sep 10 16:18:53.512614 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:18:53.512586    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9fe8919_0e96_44f7_adfe_1555e8be9959.slice/crio-1b3770aadd8bb7cfcdd1381d3750cba61328e5d9fe6902c7678fa75da3d930d8 WatchSource:0}: Error finding container 1b3770aadd8bb7cfcdd1381d3750cba61328e5d9fe6902c7678fa75da3d930d8: Status 404 returned error can't find the container with id 1b3770aadd8bb7cfcdd1381d3750cba61328e5d9fe6902c7678fa75da3d930d8
Sep 10 16:18:53.654865 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:53.654830    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68786f5998-stt8t" event={"ID":"f9fe8919-0e96-44f7-adfe-1555e8be9959","Type":"ContainerStarted","Data":"1b3770aadd8bb7cfcdd1381d3750cba61328e5d9fe6902c7678fa75da3d930d8"}
Sep 10 16:18:54.661690 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:54.661641    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68786f5998-stt8t" event={"ID":"f9fe8919-0e96-44f7-adfe-1555e8be9959","Type":"ContainerStarted","Data":"4cb6fc157967d5ddc001fa4cc0dc17543c829afb1aaac18ca4bc522752f36ca7"}
Sep 10 16:18:54.662057 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:54.661836    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-68786f5998-stt8t"
Sep 10 16:18:54.683306 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:18:54.683256    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-68786f5998-stt8t" podStartSLOduration=2.335113131 podStartE2EDuration="2.683239465s" totalImagesPullingTime="348.126334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:54.682909604 +0000 UTC m=+2213.487228866" watchObservedRunningTime="2026-09-10 16:18:54.683239465 +0000 UTC m=+2213.487558729"
Sep 10 16:19:00.672493 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:00.672466    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-68786f5998-stt8t"
Sep 10 16:19:00.728080 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:00.728045    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"]
Sep 10 16:19:00.728362 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:00.728333    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87" podUID="9e5a087a-11b0-4580-b3a2-2aef32e51516" containerName="maas-api" containerID="cri-o://3b1c7956d5e4f6ada430e50a8dcce5f1804bde034660d631ab8ac9297c1c4aa1" gracePeriod=30
Sep 10 16:19:00.985148 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:00.985123    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"
Sep 10 16:19:01.081392 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:01.081349    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5a087a-11b0-4580-b3a2-2aef32e51516-openshift-service-ca\") pod \"9e5a087a-11b0-4580-b3a2-2aef32e51516\" (UID: \"9e5a087a-11b0-4580-b3a2-2aef32e51516\") "
Sep 10 16:19:01.081598 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:01.081407    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e5a087a-11b0-4580-b3a2-2aef32e51516-metrics-certs\") pod \"9e5a087a-11b0-4580-b3a2-2aef32e51516\" (UID: \"9e5a087a-11b0-4580-b3a2-2aef32e51516\") "
Sep 10 16:19:01.081598 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:01.081513    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjhvp\" (UniqueName: \"kubernetes.io/projected/9e5a087a-11b0-4580-b3a2-2aef32e51516-kube-api-access-qjhvp\") pod \"9e5a087a-11b0-4580-b3a2-2aef32e51516\" (UID: \"9e5a087a-11b0-4580-b3a2-2aef32e51516\") "
Sep 10 16:19:01.081598 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:01.081553    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5a087a-11b0-4580-b3a2-2aef32e51516-maas-api-tls\") pod \"9e5a087a-11b0-4580-b3a2-2aef32e51516\" (UID: \"9e5a087a-11b0-4580-b3a2-2aef32e51516\") "
Sep 10 16:19:01.081799 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:01.081772    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5a087a-11b0-4580-b3a2-2aef32e51516-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9e5a087a-11b0-4580-b3a2-2aef32e51516" (UID: "9e5a087a-11b0-4580-b3a2-2aef32e51516"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:19:01.081888 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:01.081873    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5a087a-11b0-4580-b3a2-2aef32e51516-openshift-service-ca\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:19:01.083661 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:01.083631    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e5a087a-11b0-4580-b3a2-2aef32e51516-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9e5a087a-11b0-4580-b3a2-2aef32e51516" (UID: "9e5a087a-11b0-4580-b3a2-2aef32e51516"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:19:01.083782 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:01.083762    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5a087a-11b0-4580-b3a2-2aef32e51516-kube-api-access-qjhvp" (OuterVolumeSpecName: "kube-api-access-qjhvp") pod "9e5a087a-11b0-4580-b3a2-2aef32e51516" (UID: "9e5a087a-11b0-4580-b3a2-2aef32e51516"). InnerVolumeSpecName "kube-api-access-qjhvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:19:01.083850 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:01.083783    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e5a087a-11b0-4580-b3a2-2aef32e51516-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9e5a087a-11b0-4580-b3a2-2aef32e51516" (UID: "9e5a087a-11b0-4580-b3a2-2aef32e51516"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:19:01.182560 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:01.182519    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5a087a-11b0-4580-b3a2-2aef32e51516-maas-api-tls\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:19:01.182560 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:01.182552    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e5a087a-11b0-4580-b3a2-2aef32e51516-metrics-certs\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:19:01.182560 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:01.182565    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjhvp\" (UniqueName: \"kubernetes.io/projected/9e5a087a-11b0-4580-b3a2-2aef32e51516-kube-api-access-qjhvp\") on node \"ip-10-0-128-145.ec2.internal\" DevicePath \"\""
Sep 10 16:19:01.698333 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:01.698301    2564 generic.go:356] "Generic (PLEG): container finished" podID="9e5a087a-11b0-4580-b3a2-2aef32e51516" containerID="3b1c7956d5e4f6ada430e50a8dcce5f1804bde034660d631ab8ac9297c1c4aa1" exitCode=0
Sep 10 16:19:01.698793 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:01.698352    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"
Sep 10 16:19:01.698793 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:01.698389    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87" event={"ID":"9e5a087a-11b0-4580-b3a2-2aef32e51516","Type":"ContainerDied","Data":"3b1c7956d5e4f6ada430e50a8dcce5f1804bde034660d631ab8ac9297c1c4aa1"}
Sep 10 16:19:01.698793 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:01.698442    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87" event={"ID":"9e5a087a-11b0-4580-b3a2-2aef32e51516","Type":"ContainerDied","Data":"faa88ae60282c481b19ea46a9fdc877d3cf31e933baa16a8b1a4ba15e2f830e9"}
Sep 10 16:19:01.698793 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:01.698464    2564 scope.go:117] "RemoveContainer" containerID="3b1c7956d5e4f6ada430e50a8dcce5f1804bde034660d631ab8ac9297c1c4aa1"
Sep 10 16:19:01.707767 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:01.707748    2564 scope.go:117] "RemoveContainer" containerID="3b1c7956d5e4f6ada430e50a8dcce5f1804bde034660d631ab8ac9297c1c4aa1"
Sep 10 16:19:01.708086 ip-10-0-128-145 kubenswrapper[2564]: E0910 16:19:01.708068    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b1c7956d5e4f6ada430e50a8dcce5f1804bde034660d631ab8ac9297c1c4aa1\": container with ID starting with 3b1c7956d5e4f6ada430e50a8dcce5f1804bde034660d631ab8ac9297c1c4aa1 not found: ID does not exist" containerID="3b1c7956d5e4f6ada430e50a8dcce5f1804bde034660d631ab8ac9297c1c4aa1"
Sep 10 16:19:01.708179 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:01.708097    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b1c7956d5e4f6ada430e50a8dcce5f1804bde034660d631ab8ac9297c1c4aa1"} err="failed to get container status \"3b1c7956d5e4f6ada430e50a8dcce5f1804bde034660d631ab8ac9297c1c4aa1\": rpc error: code = NotFound desc = could not find container \"3b1c7956d5e4f6ada430e50a8dcce5f1804bde034660d631ab8ac9297c1c4aa1\": container with ID starting with 3b1c7956d5e4f6ada430e50a8dcce5f1804bde034660d631ab8ac9297c1c4aa1 not found: ID does not exist"
Sep 10 16:19:01.720602 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:01.720575    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"]
Sep 10 16:19:01.734105 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:01.734074    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f5df8d666-p6l87"]
Sep 10 16:19:02.897142 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:02.897112    2564 scope.go:117] "RemoveContainer" containerID="200cd1df8d0114efd78b4105b5c156716ebe4147cc6cecc6e1eb184fd689deda"
Sep 10 16:19:03.672464 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:03.672433    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e5a087a-11b0-4580-b3a2-2aef32e51516" path="/var/lib/kubelet/pods/9e5a087a-11b0-4580-b3a2-2aef32e51516/volumes"
Sep 10 16:19:23.417482 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:23.417410    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-mb7mt_69c36949-a1a5-4c1b-a250-4a08cb639da3/manager/0.log"
Sep 10 16:19:23.537557 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:23.537521    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vsdlt_ab9b3bee-e44b-4a95-a430-7a712f5c323b/manager/0.log"
Sep 10 16:19:23.660672 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:23.660637    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cd9db79db-phw5r_5ee1606b-16f3-4b8c-a77e-953d062c69e4/manager/0.log"
Sep 10 16:19:23.768795 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:23.768770    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2rkvm_f0e4d03d-ca8f-4f66-9a0f-5e1885b280f1/server/0.log"
Sep 10 16:19:24.006014 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:24.005990    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8fbcf4bcf-5vz94_a31469b0-b6aa-4281-a68a-54961f930bfd/copy-manifests/0.log"
Sep 10 16:19:24.013822 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:24.013800    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8fbcf4bcf-5vz94_a31469b0-b6aa-4281-a68a-54961f930bfd/manager/0.log"
Sep 10 16:19:24.622753 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:24.622719    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-68786f5998-stt8t_f9fe8919-0e96-44f7-adfe-1555e8be9959/maas-api/0.log"
Sep 10 16:19:24.853345 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:24.853313    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817600-6bb9m_34637ef5-80be-4d7b-ad69-cfeacc663e43/cleanup/0.log"
Sep 10 16:19:24.969831 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:24.969799    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817615-frx7b_2b3ef8b6-9a8e-4d75-b609-dc2df71220cf/cleanup/0.log"
Sep 10 16:19:25.077795 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:25.077760    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-78kbv_3869fc0f-cd42-4f52-8d4b-36a75af216d2/postgres/0.log"
Sep 10 16:19:25.961809 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:25.961781    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-csmdr_149012af-8a86-480b-b7b0-49a4cacf0aea/manager/0.log"
Sep 10 16:19:26.078780 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:26.078748    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-76pts_8479fbe4-e17d-431b-9140-1da1df8e2f64/manager/0.log"
Sep 10 16:19:26.307882 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:26.307805    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-8wmh9_bfc5bc5c-d9b9-40fb-b6a7-50711c9bcfb8/manager/0.log"
Sep 10 16:19:26.538203 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:26.538175    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-2868c_20c43a64-1f3d-4273-b3a9-3e2704dde2db/manager/0.log"
Sep 10 16:19:27.211251 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:27.211224    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6896b79569-vjk5p_5056caa3-d566-456b-935b-853c5b3be141/kube-auth-proxy/0.log"
Sep 10 16:19:27.332850 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:27.332819    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wgwhz_5d6bce58-c1f0-46be-8110-c84cab62ff75/istio-proxy/1.log"
Sep 10 16:19:27.450759 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:27.450732    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-dxfj9_8dc6aa70-fbc7-48e4-bd55-2873ea032836/payload-pre-processing/0.log"
Sep 10 16:19:27.570803 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:27.570726    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4n4q2_d551ac05-c043-4e13-90da-c8c978d709f7/payload-processing/0.log"
Sep 10 16:19:28.255068 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:28.255038    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-njtr4_204f1b8f-4b46-4c63-a48a-8f9a81d5f0a9/main/0.log"
Sep 10 16:19:28.369986 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:28.369959    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-grqxg_3a29fdc8-8361-4a32-9ce4-765dfab4f6cd/main/0.log"
Sep 10 16:19:36.746179 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:36.746140    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-znbph_8f29f7a6-d451-487d-92f1-84c47bf32d5c/global-pull-secret-syncer/0.log"
Sep 10 16:19:36.833455 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:36.833427    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q7c8n_0cad1bb8-0f30-4ca9-ad4a-fbf4f70122fb/konnectivity-agent/0.log"
Sep 10 16:19:36.904719 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:36.904687    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-145.ec2.internal_05d404cb45a3a313356b08f70399e53d/haproxy/0.log"
Sep 10 16:19:41.524399 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:41.524368    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-csmdr_149012af-8a86-480b-b7b0-49a4cacf0aea/manager/0.log"
Sep 10 16:19:41.560921 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:41.560891    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-76pts_8479fbe4-e17d-431b-9140-1da1df8e2f64/manager/0.log"
Sep 10 16:19:41.644350 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:41.644320    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-8wmh9_bfc5bc5c-d9b9-40fb-b6a7-50711c9bcfb8/manager/0.log"
Sep 10 16:19:41.741800 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:41.741766    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-2868c_20c43a64-1f3d-4273-b3a9-3e2704dde2db/manager/0.log"
Sep 10 16:19:43.639679 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:43.639648    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-6qs4r_fba202c6-588a-4976-b518-e1e95ae87c61/cluster-monitoring-operator/0.log"
Sep 10 16:19:43.681304 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:43.681279    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-cvvdl_ebaad502-c449-4e86-81b5-3c28d91d99be/kube-state-metrics/0.log"
Sep 10 16:19:43.711304 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:43.711278    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-cvvdl_ebaad502-c449-4e86-81b5-3c28d91d99be/kube-rbac-proxy-main/0.log"
Sep 10 16:19:43.740501 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:43.740420    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-cvvdl_ebaad502-c449-4e86-81b5-3c28d91d99be/kube-rbac-proxy-self/0.log"
Sep 10 16:19:43.941903 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:43.941872    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9l8f4_ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb/node-exporter/0.log"
Sep 10 16:19:43.974682 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:43.974653    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9l8f4_ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb/kube-rbac-proxy/0.log"
Sep 10 16:19:43.998704 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:43.998644    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9l8f4_ab3f1fab-bad4-4a9e-bba2-0ccf22f554bb/init-textfile/0.log"
Sep 10 16:19:44.305627 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:44.305547    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_30f064bd-9318-47a8-bb5b-ad5148b14797/prometheus/0.log"
Sep 10 16:19:44.340489 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:44.340457    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_30f064bd-9318-47a8-bb5b-ad5148b14797/config-reloader/0.log"
Sep 10 16:19:44.377375 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:44.377350    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_30f064bd-9318-47a8-bb5b-ad5148b14797/thanos-sidecar/0.log"
Sep 10 16:19:44.418402 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:44.418374    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_30f064bd-9318-47a8-bb5b-ad5148b14797/kube-rbac-proxy-web/0.log"
Sep 10 16:19:44.454451 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:44.454428    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_30f064bd-9318-47a8-bb5b-ad5148b14797/kube-rbac-proxy/0.log"
Sep 10 16:19:44.486656 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:44.486633    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_30f064bd-9318-47a8-bb5b-ad5148b14797/kube-rbac-proxy-thanos/0.log"
Sep 10 16:19:44.525767 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:44.525741    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_30f064bd-9318-47a8-bb5b-ad5148b14797/init-config-reloader/0.log"
Sep 10 16:19:44.672394 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:44.672370    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-swfht_8b39dc44-71c0-40cf-8837-dbb9a86be186/prometheus-operator-admission-webhook/0.log"
Sep 10 16:19:47.002743 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.002710    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"]
Sep 10 16:19:47.003180 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.003145    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e5a087a-11b0-4580-b3a2-2aef32e51516" containerName="maas-api"
Sep 10 16:19:47.003180 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.003176    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e5a087a-11b0-4580-b3a2-2aef32e51516" containerName="maas-api"
Sep 10 16:19:47.003271 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.003244    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e5a087a-11b0-4580-b3a2-2aef32e51516" containerName="maas-api"
Sep 10 16:19:47.006671 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.006650    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"
Sep 10 16:19:47.009117 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.009086    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2t5gt\"/\"kube-root-ca.crt\""
Sep 10 16:19:47.009263 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.009198    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2t5gt\"/\"openshift-service-ca.crt\""
Sep 10 16:19:47.010369 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.010346    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2t5gt\"/\"default-dockercfg-xmrs6\""
Sep 10 16:19:47.019548 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.019528    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"]
Sep 10 16:19:47.120605 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.120578    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-snm4z_f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f/console-operator/1.log"
Sep 10 16:19:47.125901 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.125873    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-snm4z_f0bf300c-3f8a-4e28-bbdc-21eedfd5a12f/console-operator/2.log"
Sep 10 16:19:47.140727 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.140706    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3050e480-d085-42d2-9d52-5caacb6f4815-lib-modules\") pod \"perf-node-gather-daemonset-2n22m\" (UID: \"3050e480-d085-42d2-9d52-5caacb6f4815\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"
Sep 10 16:19:47.140844 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.140746    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3050e480-d085-42d2-9d52-5caacb6f4815-sys\") pod \"perf-node-gather-daemonset-2n22m\" (UID: \"3050e480-d085-42d2-9d52-5caacb6f4815\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"
Sep 10 16:19:47.140844 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.140766    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3050e480-d085-42d2-9d52-5caacb6f4815-podres\") pod \"perf-node-gather-daemonset-2n22m\" (UID: \"3050e480-d085-42d2-9d52-5caacb6f4815\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"
Sep 10 16:19:47.140919 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.140877    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3050e480-d085-42d2-9d52-5caacb6f4815-proc\") pod \"perf-node-gather-daemonset-2n22m\" (UID: \"3050e480-d085-42d2-9d52-5caacb6f4815\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"
Sep 10 16:19:47.140919 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.140913    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn29j\" (UniqueName: \"kubernetes.io/projected/3050e480-d085-42d2-9d52-5caacb6f4815-kube-api-access-cn29j\") pod \"perf-node-gather-daemonset-2n22m\" (UID: \"3050e480-d085-42d2-9d52-5caacb6f4815\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"
Sep 10 16:19:47.241986 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.241952    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn29j\" (UniqueName: \"kubernetes.io/projected/3050e480-d085-42d2-9d52-5caacb6f4815-kube-api-access-cn29j\") pod \"perf-node-gather-daemonset-2n22m\" (UID: \"3050e480-d085-42d2-9d52-5caacb6f4815\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"
Sep 10 16:19:47.242205 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.242002    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3050e480-d085-42d2-9d52-5caacb6f4815-lib-modules\") pod \"perf-node-gather-daemonset-2n22m\" (UID: \"3050e480-d085-42d2-9d52-5caacb6f4815\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"
Sep 10 16:19:47.242205 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.242048    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3050e480-d085-42d2-9d52-5caacb6f4815-sys\") pod \"perf-node-gather-daemonset-2n22m\" (UID: \"3050e480-d085-42d2-9d52-5caacb6f4815\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"
Sep 10 16:19:47.242205 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.242076    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3050e480-d085-42d2-9d52-5caacb6f4815-podres\") pod \"perf-node-gather-daemonset-2n22m\" (UID: \"3050e480-d085-42d2-9d52-5caacb6f4815\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"
Sep 10 16:19:47.242205 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.242121    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3050e480-d085-42d2-9d52-5caacb6f4815-sys\") pod \"perf-node-gather-daemonset-2n22m\" (UID: \"3050e480-d085-42d2-9d52-5caacb6f4815\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"
Sep 10 16:19:47.242205 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.242152    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3050e480-d085-42d2-9d52-5caacb6f4815-proc\") pod \"perf-node-gather-daemonset-2n22m\" (UID: \"3050e480-d085-42d2-9d52-5caacb6f4815\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"
Sep 10 16:19:47.242205 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.242198    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3050e480-d085-42d2-9d52-5caacb6f4815-lib-modules\") pod \"perf-node-gather-daemonset-2n22m\" (UID: \"3050e480-d085-42d2-9d52-5caacb6f4815\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"
Sep 10 16:19:47.242529 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.242223    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3050e480-d085-42d2-9d52-5caacb6f4815-proc\") pod \"perf-node-gather-daemonset-2n22m\" (UID: \"3050e480-d085-42d2-9d52-5caacb6f4815\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"
Sep 10 16:19:47.242529 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.242260    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3050e480-d085-42d2-9d52-5caacb6f4815-podres\") pod \"perf-node-gather-daemonset-2n22m\" (UID: \"3050e480-d085-42d2-9d52-5caacb6f4815\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"
Sep 10 16:19:47.252077 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.252054    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn29j\" (UniqueName: \"kubernetes.io/projected/3050e480-d085-42d2-9d52-5caacb6f4815-kube-api-access-cn29j\") pod \"perf-node-gather-daemonset-2n22m\" (UID: \"3050e480-d085-42d2-9d52-5caacb6f4815\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"
Sep 10 16:19:47.316508 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.316422    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"
Sep 10 16:19:47.591926 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.591894    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"]
Sep 10 16:19:47.594550 ip-10-0-128-145 kubenswrapper[2564]: W0910 16:19:47.594516    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3050e480_d085_42d2_9d52_5caacb6f4815.slice/crio-b9ec8cadbe08b37244e2bcb277c81d43ade5cb561131533d136f12d0f4a4c856 WatchSource:0}: Error finding container b9ec8cadbe08b37244e2bcb277c81d43ade5cb561131533d136f12d0f4a4c856: Status 404 returned error can't find the container with id b9ec8cadbe08b37244e2bcb277c81d43ade5cb561131533d136f12d0f4a4c856
Sep 10 16:19:47.753486 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.753459    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6bbc679795-dzhmg_3989901e-a627-4be6-bbde-dfa31a067064/console/0.log"
Sep 10 16:19:47.791560 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.791532    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-qlzmm_b8a041bb-e51d-4b49-92c8-e25b58a05f3a/download-server/0.log"
Sep 10 16:19:47.947534 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.947497    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m" event={"ID":"3050e480-d085-42d2-9d52-5caacb6f4815","Type":"ContainerStarted","Data":"faade85d9fc69a8f26c956b934434d3a9594fa9132e92a3d30729cd65e2bf8b7"}
Sep 10 16:19:47.947703 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.947555    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m" event={"ID":"3050e480-d085-42d2-9d52-5caacb6f4815","Type":"ContainerStarted","Data":"b9ec8cadbe08b37244e2bcb277c81d43ade5cb561131533d136f12d0f4a4c856"}
Sep 10 16:19:47.947703 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.947659    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"
Sep 10 16:19:47.982193 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:47.982127    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m" podStartSLOduration=1.982111792 podStartE2EDuration="1.982111792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:19:47.980449755 +0000 UTC m=+2266.784769021" watchObservedRunningTime="2026-09-10 16:19:47.982111792 +0000 UTC m=+2266.786431092"
Sep 10 16:19:48.538901 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:48.538870    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-g45nr_14920648-4049-47e8-983a-8b9d7f41182b/volume-data-source-validator/0.log"
Sep 10 16:19:49.630459 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:49.630432    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2qrbr_db9487f8-8adc-4d2e-8c5b-7222bf915898/dns/0.log"
Sep 10 16:19:49.656231 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:49.656197    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2qrbr_db9487f8-8adc-4d2e-8c5b-7222bf915898/kube-rbac-proxy/0.log"
Sep 10 16:19:49.846306 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:49.846272    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hb6s4_c5fe8c6d-b8ce-4138-b5ed-a05499ccd346/dns-node-resolver/0.log"
Sep 10 16:19:50.411139 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:50.411113    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5cf899df5c-trxv8_eae7786c-775a-4130-b011-415e5c1e18b8/registry/0.log"
Sep 10 16:19:50.489307 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:50.489281    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k6fhq_e6f2f68d-170c-4567-ba06-b05bea2cb642/node-ca/0.log"
Sep 10 16:19:51.757735 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:51.757701    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6896b79569-vjk5p_5056caa3-d566-456b-935b-853c5b3be141/kube-auth-proxy/0.log"
Sep 10 16:19:51.803365 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:51.803337    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wgwhz_5d6bce58-c1f0-46be-8110-c84cab62ff75/istio-proxy/0.log"
Sep 10 16:19:51.812768 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:51.812745    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wgwhz_5d6bce58-c1f0-46be-8110-c84cab62ff75/istio-proxy/1.log"
Sep 10 16:19:51.853965 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:51.853935    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-dxfj9_8dc6aa70-fbc7-48e4-bd55-2873ea032836/payload-pre-processing/0.log"
Sep 10 16:19:51.902511 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:51.902482    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4n4q2_d551ac05-c043-4e13-90da-c8c978d709f7/payload-processing/0.log"
Sep 10 16:19:52.847474 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:52.847443    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8bc5c_28343b32-70a8-4d77-8dfd-d7b27a3f8cef/serve-healthcheck-canary/0.log"
Sep 10 16:19:53.382862 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:53.382833    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-kk7mn_21f9ed82-8d7f-48dc-af6f-a22d73178304/insights-operator/0.log"
Sep 10 16:19:53.384790 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:53.384774    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-kk7mn_21f9ed82-8d7f-48dc-af6f-a22d73178304/insights-operator/1.log"
Sep 10 16:19:53.569172 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:53.569128    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pr6k5_7d601040-aa12-4aab-85df-a2c78ddc3f68/kube-rbac-proxy/0.log"
Sep 10 16:19:53.592402 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:53.592377    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pr6k5_7d601040-aa12-4aab-85df-a2c78ddc3f68/exporter/0.log"
Sep 10 16:19:53.617730 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:53.617697    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pr6k5_7d601040-aa12-4aab-85df-a2c78ddc3f68/extractor/0.log"
Sep 10 16:19:53.964297 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:53.964270    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-2n22m"
Sep 10 16:19:55.830613 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:55.830587    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-mb7mt_69c36949-a1a5-4c1b-a250-4a08cb639da3/manager/0.log"
Sep 10 16:19:56.010896 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:56.010870    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vsdlt_ab9b3bee-e44b-4a95-a430-7a712f5c323b/manager/0.log"
Sep 10 16:19:56.062584 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:56.062557    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cd9db79db-phw5r_5ee1606b-16f3-4b8c-a77e-953d062c69e4/manager/0.log"
Sep 10 16:19:56.105700 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:56.105668    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2rkvm_f0e4d03d-ca8f-4f66-9a0f-5e1885b280f1/server/0.log"
Sep 10 16:19:56.233529 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:56.233491    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8fbcf4bcf-5vz94_a31469b0-b6aa-4281-a68a-54961f930bfd/manager/0.log"
Sep 10 16:19:56.260889 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:56.260862    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8fbcf4bcf-5vz94_a31469b0-b6aa-4281-a68a-54961f930bfd/copy-manifests/0.log"
Sep 10 16:19:57.924845 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:57.924821    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6896b79569-vjk5p_5056caa3-d566-456b-935b-853c5b3be141/kube-auth-proxy/0.log"
Sep 10 16:19:57.947711 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:57.947683    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wgwhz_5d6bce58-c1f0-46be-8110-c84cab62ff75/istio-proxy/0.log"
Sep 10 16:19:57.955641 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:57.955610    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wgwhz_5d6bce58-c1f0-46be-8110-c84cab62ff75/istio-proxy/1.log"
Sep 10 16:19:57.980030 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:57.979998    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-dxfj9_8dc6aa70-fbc7-48e4-bd55-2873ea032836/payload-pre-processing/0.log"
Sep 10 16:19:58.007638 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:58.007612    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4n4q2_d551ac05-c043-4e13-90da-c8c978d709f7/payload-processing/0.log"
Sep 10 16:19:59.820937 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:59.820913    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6896b79569-vjk5p_5056caa3-d566-456b-935b-853c5b3be141/kube-auth-proxy/0.log"
Sep 10 16:19:59.842453 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:59.842428    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wgwhz_5d6bce58-c1f0-46be-8110-c84cab62ff75/istio-proxy/0.log"
Sep 10 16:19:59.850522 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:59.850500    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wgwhz_5d6bce58-c1f0-46be-8110-c84cab62ff75/istio-proxy/1.log"
Sep 10 16:19:59.878147 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:59.878122    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-dxfj9_8dc6aa70-fbc7-48e4-bd55-2873ea032836/payload-pre-processing/0.log"
Sep 10 16:19:59.912795 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:19:59.912766    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4n4q2_d551ac05-c043-4e13-90da-c8c978d709f7/payload-processing/0.log"
Sep 10 16:20:03.523773 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:20:03.523742    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-nsnkp_811b2e5a-e4b6-45d9-99d4-4669a476afbc/kube-storage-version-migrator-operator/1.log"
Sep 10 16:20:03.524576 ip-10-0-128-145 kubenswrapper[2564]: I0910 16:20:03.524555    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-nsnkp_811b2e5a-e4b6-45d9-99d4-4669a476afbc/kube-storage-version-migrator-operator/0.log"